RoSHI: A Versatile Robot-oriented Suit for Human Data In-the-Wild

RoSHI combines nine low-cost IMU trackers (~$30 each, BNO085-based, wireless at 100 Hz) with Project Aria glasses to capture synchronized 3D body pose, egocentric RGB video, and a globally consistent root trajectory. The two modalities are complementary: IMUs provide robustness to visual occlusion and high-speed motion, while egocentric SLAM anchors long-horizon global localization and stabilizes upper-body pose.

Calibration requires only a short 20–40 second video captured by an iPhone while wearing the suit. Each tracker has a rigidly mounted AprilTag; combined with SAM 3D Body estimates from the calibration video, this recovers sensor-to-bone and cross-sensor heading alignments without a box calibration or prescribed poses. It enables quick recalibration at any time without removing the IMUs.

For body pose generation, we leverage the EgoAllo diffusion model conditioned on the 6-DoF headset trajectory from Aria SLAM. We guide diffusion using bone orientations derived from the IMU trackers and enforce three complementary constraints: (i) direct comparison of observable joint angles (elbow, hip, knee), (ii) relative orientation consistency between the pelvis and shoulders, and (iii) temporal smoothness of pelvis-joint rotations across consecutive frames.

We evaluate RoSHI on 11 motion sequences across three datasets covering in-place motions, locomotion with global translation, and agile sports-like activities. RoSHI achieves the best mean per-joint position error (MPJPE) across all three datasets and the best joint angle error (JAE) on two of three, showing consistent improvements in both global joint localization and articulated pose reconstruction over egocentric baselines.

TODO: Add acknowledgements here.