Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
HPT: per-robot stems turn proprioception and vision into a few tokens for one shared transformer trunk, pretrained on 52 datasets (robot, sim, human video); reports 20%+ gains on unseen tasks.HPT：为每种机器人设计“输入头”，把本体感知与视觉映射成短 token 序列，送入共享的 Transformer 主干；在 52 个数据集（真机、仿真、人类视频）上预训练，报告在未见任务上微调提升 20% 以上。
BibTeX
@misc{wang2024scaling,
title = {{Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers}},
author = {Lirui Wang and Xinlei Chen and Jialiang Zhao and Kaiming He},
year = {2024},
eprint = {2409.20537},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2409.20537},
}VLA & Robot Foundation ModelsVLA 与机器人基础模型Robot Learning: RL & Imitation机器人学习：强化学习与模仿学习realsimcross-embodimentpretrainingopen-weights