Predicting touch from egocentric human video for contact-rich robot manipulation
Ego2Contact learns a visual–tactile prior from human egocentric video paired with tactile-glove measurements, transfers it to robot hands through anatomical region correspondences, and uses the predicted touch as an observation for reinforcement learning in simulation and for vision-language-action post-training on a real dexterous hand.
A visual–tactile predictor is learned from paired human egocentric video and tactile measurements, fine-tuned to a dexterous robot hand through anatomical region correspondences, and used to supply touch for policy learning across four simulated and four real-world contact-rich tasks.
Unscrewing a cap, inserting a tube into a rack or pouring between two hands depends on where contact occurs and how force evolves. A static image or hand pose does not determine the pressure underneath.
Tactile skins and optical sensors are hard to integrate and tied to specific end-effectors, so tactile recordings are costly, rarely reused across platforms, and largely absent from cross-embodiment datasets.
Egocentric human video is easy to collect. Recent glove datasets pair it with dense full-hand pressure, yet existing predictors stay in the human domain and do not drive a robot policy.
Can a tactile prior learned from human video supply explicit touch observations that robot policies can consume?
Ego2Contact formulates tactile estimation as conditional flow-matching generation over a canonical 21×21 hand grid, and uses anatomical region correspondences on that grid as the interface between human supervision, robot sensors, and policy inputs.
On EgoTouch, Ego2Contact outperforms the TouchAnything baseline and a pose-conditioned variant on seen and unseen objects. Hand-local visual conditioning matters most when the object is new, and the advantage carries over to a second dataset, EgoTactile.
| Method | Seen objects | Unseen objects | ||||||
|---|---|---|---|---|---|---|---|---|
| C.IoU ↑ | V.IoU ↑ | T.Acc ↑ | MAE ↓ | C.IoU ↑ | V.IoU ↑ | T.Acc ↑ | MAE ↓ | |
| TouchAnything | 0.5064 | 0.4260 | 0.8899 | 0.0485 | 0.4720 | 0.3457 | 0.8382 | 0.0641 |
| Ego2Contact (pose) | 0.5657 | 0.4899 | 0.9014 | 0.0385 | 0.4441 | 0.3183 | 0.8122 | 0.0596 |
| Ego2Contact | 0.5735 | 0.4961 | 0.9018 | 0.0383 | 0.4812 | 0.3417 | 0.8544 | 0.0590 |
| Method | C.IoU ↑ | V.IoU ↑ | T.Acc ↑ | MAE ↓ |
|---|---|---|---|---|
| Ego2Contact (pose) | 0.6414 | 0.4557 | 0.8520 | 0.0352 |
| Ego2Contact | 0.6988 | 0.5453 | 0.9209 | 0.0304 |
C.IoU: thresholded contact overlap · V.IoU: pressure-volume overlap · T.Acc: frame-level contact presence · MAE: normalised pressure error on valid sensor cells.
Each clip shows the model's RGB input (left) and the 21×21 pressure grids of both hands (right): ground truth on top, Ego2Contact prediction below. Pressure is normalised per sequence. Clips play while in view; click one to pause or resume it, or press ⤢ to enlarge it.
Ground-truth-tactile policies are rolled out in Isaac Gym to collect paired RGB and rigid-body contact forces. Fine-tuning on 8,741 clips from 2,258 episodes raises grid contact IoU on held-out robot episodes from 0.008 (human-trained checkpoint) to 0.476, with better contact placement than the adapted TouchAnything baseline.
| Method | C.IoU ↑ | V.IoU ↑ | MAE ↓ | T.Acc ↑ | Link C.IoU ↑ | Link CoP ↓ |
|---|---|---|---|---|---|---|
| TouchAnything | 0.415 | 0.384 | 0.0025 | 0.992 | 0.520 | 0.300 |
| Ego2Contact | 0.476 | 0.440 | 0.0025 | 0.991 | 0.571 | 0.224 |
Browse held-out simulator episodes of the four tasks frame by frame. For each hand we show ground truth and prediction in the 21×21 canonical grid and on the hand surface, and pooled over the 17 link regions that the RL policy consumes. Values are normalised per task.
Grid: the right hand is drawn mirrored (thumb on the left); each grid column lines up with the finger drawn on the hand. Hand: canonical flat hand, each link coloured by its pooled value; links below 0.03 stay black. Episodes shown were not used for predictor fine-tuning gradients. ← / → keys step frames.
Four contact-rich dexterous tasks in Isaac Gym: Pen Cap Removal (built on Bi-DexHands) and Lever Sliding, Bottle Cap Turning and Bimanual Handover (adapted from VTDexManip). Every PPO policy receives the proprioceptive base observation (robot state and task-required state); the two tactile conditions append online predicted or ground-truth tactile in a matched 17-link representation (17 pressure values and 17 contact flags per hand).
Successful executions of the four simulated tasks (episodes from the ground-truth-tactile rollouts collected for predictor adaptation).
| Task | P | P + Pred-Tac | P + GT-Tac |
|---|---|---|---|
| Pen cap removal | 0.0 | 1.0 | 95.0 |
| Lever sliding | 84.0 | 88.8 | 98.2 |
| Bottle cap turning | 64.1 | 71.0 | 74.1 |
| Bimanual handover | 29.3 | 33.1 | 49.1 |
Predicted tactile improves over the proprioception-only baseline on three of four tasks and approaches ground-truth tactile on bottle-cap turning.
* P: proprioception (base observation) · Pred-Tac: online predicted tactile · GT-Tac: ground-truth tactile.
A single-arm platform with a dexterous hand wearing a 15-pad, 880-taxel glove, a head RGB-D camera and a wrist camera. Demonstrations are collected by VR teleoperation; the human-trained predictor is adapted on about 123k paired robot frames and then annotates demonstration videos with tactile images used to post-train a vision-language-action policy. We post-train three variants that differ only in the tactile image stack the policy sees during post-training: zero-valued tactile (the tactile input is all zeros, i.e. no touch information), predicted tactile (tactile images generated by the adapted Ego2Contact predictor from the demonstration videos; ours), and measured tactile (the glove readings recorded during the demonstrations). During rollouts every policy receives the glove's measured tactile.
Average success rate over the four tasks
| Task | Zero | Pred | GT |
|---|---|---|---|
| Bottle cap removal | 5/10 | 7/10 | 7/10 |
| Test tube transfer | 3/10 | 5/10 | 6/10 |
| Whiteboard erasing | 2/10 | 3/10 | 4/10 |
| Bimanual pouring | 1/10 | 3/10 | 5/10 |
Zero: zero-valued tactile · Pred: predicted tactile (ours) · GT: measured (glove) tactile, all referring to the tactile input used during policy post-training.
Head-camera view with the wrist camera(s) inset at the top. These are individual qualitative rollouts, not aggregate success-rate estimates.