Comparison of Multiagent Learning Algorithms in Ad Hoc Teams

Multiagent Learning (MAL) is the algorithmic study of learning in a group of two or more agents. If the agents are based on different algorithms, and if there is no form of prior coordination between the agents, then this is called an ad hoc team problem [48]. Following a literature review [2] and a research proposal [1], the work at hand compares the performance of five MAL algorithms in ad hoc teams. These include the Joint Action Learner [12], the Conditional Joint Action Learner [3], Win or Learn Fast with Policy Hill Climbing [6], Modified Regret-Matching [21], and the Nash Q-Learner [24]. The algorithms are evaluated in a range of strategic games, including no-conflict games in which the players agree on what is most preferred, and conflict games in which the players disagree on what is most preferred [40]. In addition, we use an evaluation procedure proposed by Stone et al. [48]. Our performance criteria include the convergence rate, the final expected payoff, social welfare and fairness, and the rates of different solution types. From the results we conclude that (a) all algorithms perform well in some sense (i.e., there is no clear winner), and (b) the performance of an algorithm ultimately depends on the solution concept that is considered most appropriate for the ad hoc team problem at hand.