HUI360: A dataset and baselines for Human Robot Interaction Anticipation

As robots increasingly operate in human-populated environments, anticipating human intentions is essential for enabling proactive and socially aware behavior. Automatic anticipation of human–robot interactions is thus emerging as a crucial perception challenge for embodied agents.

To this end, we introduce HUI360, the largest dataset for human-robot interaction anticipation in the wild and its set of baselines. The dataset was collected from a mobile robot, in the wild, over multiple days within a 3-month period, and in several environments, capturing natural, spontaneous behaviors from both passersby and users, and encompassing a diverse range of individuals. This variety enables evaluating and improving the generalization capabilities of interaction anticipation models.

We designed a pipeline and share code for automatic interaction annotation in arbitrary 360° equirectangular videos, along with interfaces for manual refinement. Using this pipeline, we release the HUI360 open set of 1M pre-processed annotations, including detailed 2D poses, facial keypoints, and segmentation masks, obtained using state-of-the-art computer vision methods and manually curated to ensure high-quality tracking and interaction annotation. Additionally, we release the raw panoptic 360° images captured from the robot’s egocentric viewpoint (on demand, for research purpose only in compliance with GDPR).

Finally, we establish benchmark baselines for interaction anticipation, including the first cross-dataset evaluations for this task: to this end, we also release 6M annotations for another existing in-the-wild outdoor dataset collected from a mobile robot (SSUP-HRI).

You can freely access the processed annotations (skeletons, keypoints, masks, etc.) on HUI360.

Access to the raw 360° videos from our Shelfy recordings requires a Data Transfer Agreement (DTA). These DTAs cover our HUI360 recordings only — not the SSUP-HRI videos, which are subject to their own agreements (see the SSUP-HRI section). Access is reserved for researchers at academic institutions, for research purposes only, and subject to GDPR compliance.

To request access to the raw videos:

1. Download and fill the appropriate DTA for your country of affiliation:
   • DTA — EEA and adequate countries
   • DTA — outside the EEA / non-adequate countries

   Note: As of October 16, 2023, institutes located in the following territories may use the EEA / adequate-countries form: countries of the European Union, Iceland, Liechtenstein, Norway, Andorra, Argentina, Canada, Faroe Islands, Guernsey, Israel, Isle of Man, Japan, Jersey, New Zealand, Switzerland, Uruguay, South Korea, and the United Kingdom. All other affiliations should use the outside-EEA form.

2. Sign it using your academic / institutional email address (If you are not a permanent researcher ask your supervisor to do so and add you in the "Additional researchers with access to the dataset" section).
3. Send the completed form to huidataset@inria.fr.
4. Wait for approval from our legal team.
5. In the meantime, register on Hugging Face with the same academic email and request access to HUI360-Videos. Access will be granted once the DTA is approved.