Learning from human preferences is essential for aligning machine learning models with subjective judgments, but collecting preference data is costly. We study a hybrid framework that combines the scalability of Reinforcement Learning from Human Feedback (RLHF), which trains neural reward models from pairwise comparisons, with the sample efficiency of Preferential Bayesian Optimization. The method integrates Laplace-based Bayesian uncertainty estimation to guide informative preference queries. On high-dimensional Rosenbrock optimization, the approach successfully converges in problems with up to 50 dimensions. In large language model (LLM) fine-tuning, it improves reward-model accuracy by a value within 6–14% under limited annotation budgets.
Efficient Reinforcement Learning from Human Feedback via Bayesian preference inference
Cercola M.;Capretti V.;Formentin S.
2026-01-01
Abstract
Learning from human preferences is essential for aligning machine learning models with subjective judgments, but collecting preference data is costly. We study a hybrid framework that combines the scalability of Reinforcement Learning from Human Feedback (RLHF), which trains neural reward models from pairwise comparisons, with the sample efficiency of Preferential Bayesian Optimization. The method integrates Laplace-based Bayesian uncertainty estimation to guide informative preference queries. On high-dimensional Rosenbrock optimization, the approach successfully converges in problems with up to 50 dimensions. In large language model (LLM) fine-tuning, it improves reward-model accuracy by a value within 6–14% under limited annotation budgets.| File | Dimensione | Formato | |
|---|---|---|---|
|
1-s2.0-S2468601826000386-main.pdf
accesso aperto
Dimensione
875.62 kB
Formato
Adobe PDF
|
875.62 kB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.



