Empirical Bayesian Multi-Bandit Learning

Abstract

Multi-task learning in contextual bandits has attracted significant research interest due to its potential to enhance decisionmaking across multiple related tasks by leveraging shared structures while accommodating task-specific heterogeneity. In this article, we propose a novel hierarchical Bayesian framework for learning across multiple bandit instances. This framework captures both heterogeneity and dependence among bandit instances through a hierarchical Bayesian model, enabling effective information sharing while accommodating instance-specific variations. Unlike existing methods that overlook the learning of the covariance structure across bandits, we introduce an empirical Bayesian approach to estimate the covariance matrix of the prior distribution. This enhances the practicality and flexibility of learning across multi-bandits. Building on this approach, we develop two efficient algorithms, ebmTS (empirical Bayesian multi-bandit Thompson Sampling) and ebmUCB (empirical Bayesian multi-bandit Upper Confidence Bound), which incorporate the estimated prior into the decision-making process. We establish the frequentist regret upper bounds for the proposed algorithms, tproviding theoretical guarantees for learning across multiple bandit instances. Extensive experiments on synthetic and real-world datasets demonstrate the superior performance of our proposed methods, particularly in complex environments. Our algorithms achieve lower cumulative regret compared to existing approaches across a range of settings, highlighting their ability to effectively balance exploration and exploitation while leveraging shared information across bandits.