What Are We Actually Benchmarking in Robot Manipulation?
CoRL 2026 Oral. Audits LIBERO, CALVIN, SimplerEnv, RoboCasa and RoboTwin 2.0 against four failure modes, from shortcut solvability to creeping overfitting. LIBERO and CALVIN fail several diagnostics.CoRL 2026 Oral。用捷径可解、统计不显著、渐进过拟合、数据来源依赖四种失效模式审计 LIBERO、CALVIN、SimplerEnv、RoboCasa 和 RoboTwin 2.0；其中 LIBERO 与 CALVIN 未通过多项诊断。
BibTeX
@misc{jiang2026what,
title = {{What Are We Actually Benchmarking in Robot Manipulation?}},
author = {Tianchong Jiang and Xiangshan Tan and Samuel Wheeler and Luzhe Sun and Tewodros W. Ayalew and Matthew Walter},
year = {2026},
eprint = {2606.04233},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2606.04233},
}