EgoExplorer records 100 hours of human work for robotics research

Robotics researchers looking for human demonstrations face a choice between portable recording and knowing precisely what a person’s body and hands did. In an October 13 preprint, researchers led by Shanghai AI Laboratory describe EgoExplorer, a wearable capture system and a collection of 2,715 recordings totaling 100 hours. It combines first-person video, five body trackers, instrumented gloves and, for recordings made with one glove configuration, touch readings. The team also reconstructed full-body motion from the sparse tracker signals.

That last distinction matters when deciding how to use the data. A tracked ankle is an observation; an elbow trajectory between tracked points is a model-informed reconstruction. The paper tests reconstruction against fitted reference labels and replays two recorded movements with a simulated humanoid. It does not show a physical robot learning to perform the recorded jobs.

The big change

• What changed: EgoExplorer brings camera, body-tracker, finger and available touch data onto a shared timeline across 100 hours of recorded activity. Its full-body poses are reconstructed from five trackers, so researchers can inspect the measurements separately from the inferred joints.
• Why it matters: A team selecting demonstrations for mobile manipulation can study walking, posture and hand interaction in the same episode. That is more informative for choosing training inputs than a hand-only clip, provided the team accounts for reconstruction error and gaps in touch coverage.
• What to watch: The present dataset release is a gated 66-minute preview; the larger collection is being released in stages. Its value for robot training still depends on tests across unseen tasks and physical hardware. The paper’s two simulated replay clips do not settle either question.

What the wearable actually records

The paper’s hardware description places five inside-out trackers at the head, wrists and ankles, alongside a head-mounted camera. MANUS gloves measure finger articulation; the alternative Wuji glove configuration adds tactile readings. Each stream is aligned by timestamp, and the resulting recordings include step descriptions that operators review after an automated annotation pass. Ten task templates span five environments, including laboratory, industrial and home settings.

Those devices do not directly observe every joint. The team’s Calibrate-Complete-Constrain pipeline first estimates each participant’s body shape, scale and sensor attachments from prescribed movements. A temporal motion model proposes the unobserved joint positions. Further fitting presses the proposed body toward tracker measurements while penalizing implausible self-intersection and penetration of the floor. Glove-derived finger motion and available touch data are then associated with that reconstructed body. The paper explicitly says exact recovery of every joint cannot be guaranteed and tactile reliability varies.

What the accuracy number measures

On a controlled evaluation subset, EgoExplorer reports 18.0 mm body-point error, compared with 64.1 mm for the strongest body-point baseline in its table, a 71.9% reduction. Both figures are distances from tracker-fitted reference labels. The references were produced using the same five trackers, participant calibration and a more intensive fitting process with a pose prior; they are not an independent optical motion-capture ground truth. The test portion consisted of ten held-out recordings from a controlled set of body actions, after earlier repetitions supported adaptation of the proposal network.

That result supports a narrower decision than the percentage alone suggests: in this matched setup, the method reproduced its fitted body references more closely than the evaluated alternatives. Several baselines used three trackers, while the closest five-tracker AvatarPoser variant had its own training and adaptation. The rest of the natural recordings have no matched accuracy reference. The paper separately evaluates held hand poses and regional contact, with different references and exclusions; those scores cannot be read as physical task success.

A simulated robot replay, with a narrow scope

For an initial robot study, the researchers converted two recorded human clips—reaction setup and reagent weighing—into targets for a Unitree G1 model. A reinforcement-learning controller trained on those two clips tracked the resulting robot references in MuJoCo simulation. The appendix reports no falls during full-clip replay, but also reports centimeter-scale body and wrist deviations and says the results do not demonstrate precise object interaction. The fingers were replayed kinematically rather than controlled by the body policy.

The controller was evaluated on the clips it trained on. This is evidence that the reconstructed motions could be retargeted and tracked in that simulation, under those conditions. It does not establish transfer to an unseen job or safe operation on a physical G1. The authors identify state estimation and safety constraints as additional requirements for hardware deployment.

Researchers can inspect the released toolkit and request access to the gated dataset preview. The preview contains 18 episodes, about 66 minutes, across all ten task templates; access to the full collection is staged through a waitlist. The dataset card lists a CC BY-NC 4.0 license, distinct from the arXiv page’s CC BY-NC-SA 4.0 paper license. A prospective user should check whether the available episodes, measured streams, reconstruction assumptions and noncommercial terms suit the intended research.