Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
Keeps conservative offline Q-values from dropping below a reference policy's value, which avoids the early performance dip when an offline-pretrained policy starts online fine-tuning.约束保守的离线 Q 值不低于参考策略的价值，从而避免离线预训练策略开始在线微调时常见的性能先降后升。
BibTeX
@misc{nakamoto2023cal,
title = {{Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning}},
author = {Mitsuhiko Nakamoto and Yuexiang Zhai and Anikait Singh and Max Sobol Mark and Yi Ma and Chelsea Finn and Aviral Kumar and Sergey Levine},
year = {2023},
eprint = {2303.05479},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2303.05479},
}