Basins of attraction and equilibrium selection under different learning rules

A deterministic learning model applied to a game with multiple equilibria produces distinct basins of attraction for those equilibria. In symmetric two-by-two games, basins of attraction are invariant to a wide range of learning rules including best response dynamics, replicator dynamics, and fictitious play. In this paper, we construct a class of three-by-three symmetric games for which the overlap in the basins of attraction under best response learning and replicator dynamics is arbitrarily small. We then derive necessary and sufficient conditions on payoffs for these two learning rules to create basins of attraction with vanishing overlap. The necessary condition requires that with probability one the initial best response is not an equilibrium to the game. The existence of parasitic or misleading actions allows subtle differences in the learning rules to accumulate.     

Generalised weakened fictitious play

A general class of adaptive processes in games is developed, which significantly generalises weakened fictitious play [Van der Genugten, B., 2000. A weakened form of fictitious play in two-person zero-sum games. Int. Game Theory Rev. 2, 307–328] and includes several interesting fictitious-play-like processes as special cases. The general model is rigorously analysed using the best response differential inclusion, and shown to converge in games with the fictitious play property. Furthermore, a new actor–critic process is introduced, in which the only information given to a player is the reward received as a result of selecting an action—a player need not even know they are playing a game. It is shown that this results in a generalised weakened fictitious play process, and can therefore be considered as a first step towards explaining how players might learn to play Nash equilibrium strategies without having any knowledge of the game, or even that they are playing a game.     

Self-tuning experience weighted attraction learning in games

Self-tuning experience weighted attraction (EWA) is a one-parameter theory of learning in games. It addresses a criticism that an earlier model (EWA) has too many parameters, by fixing some parameters at plausible values and replacing others with functions of experience so that they no longer need to be estimated. Consequently, it is econometrically simpler than the popular weighted fictitious play and reinforcement learning models. The functions of experience which replace free parameters “self-tune” over time, adjusting in a way that selects a sensible learning rule to capture subjects’ choice dynamics. For instance, the self-tuning EWA model can turn from a weighted fictitious play into an averaging reinforcement learning as subjects equilibrate and learn to ignore inferior foregone payoffs. The theory was tested on seven different games, and compared to the earlier parametric EWA model and a one-parameter stochastic equilibrium theory (QRE). Self-tuning EWA does as well as EWA in predicting behavior in new games, even though it has fewer parameters, and fits reliably better than the QRE equilibrium benchmark.     

Learning from Personal Experience: One Rational Guy and the Justification of Myopia

The paper examines a large population analog of fictitious play in which players learn from personal experience, focusing on what happens when a single rational player is added to the population. Because the learning process naturally generates contagion dynamics, the rational player at times has an incentive to act nonmyopically. In 2 × 2 games the dynamics are asymmetric and favor risk dominant equilibria. A variety of other examples are presented.Journal of Economic LiteratureClassification Number: C7.     

Brown's original fictitious play

What modern game theorists describe as “fictitious play” is not the learning process George W. Brown defined in his 1951 paper. Brown's original version differs in a subtle detail, namely the order of belief updating. In this note we revive Brown's original fictitious play process and demonstrate that this seemingly innocent detail allows for an extremely simple and intuitive proof of convergence in an interesting and large class of games: nondegenerate ordinal potential games.     

Learning in games with unstable equilibria

We propose a new concept for the analysis of games, the TASP, which gives a precise prediction about non-equilibrium play in games whose Nash equilibria are mixed and are unstable under fictitious play-like learning. We show that, when players learn using weighted stochastic fictitious play and so place greater weight on recent experience, the time average of play often converges in these “unstable” games, even while mixed strategies and beliefs continue to cycle. This time average, the TASP, is related to the cycle identified by Shapley [L.S. Shapley, Some topics in two person games, in: M. Dresher, et al. (Eds.), Advances in Game Theory, Princeton University Press, Princeton, 1964]. The TASP can be close to or quite distinct from Nash equilibrium.     

Dynamics in near-potential games

We consider discrete-time learning dynamics in finite strategic form games, and show that games that are close to a potential game inherit many of the dynamical properties of potential games. We first study the evolution of the sequence of pure strategy profiles under better/best response dynamics. We show that this sequence converges to a (pure) approximate equilibrium set whose size is a function of the “distance” to a given nearby potential game. We then focus on logit response dynamics, and provide a characterization of the limiting outcome in terms of the distance of the game to a given potential game and the corresponding potential function. Finally, we turn attention to fictitious play, and establish that in near-potential games the sequence of empirical frequencies of player actions converges to a neighborhood of (mixed) equilibria, where the size of the neighborhood increases according to the distance to the set of potential games.     

The rate of convergence of continuous fictitious play★

Summary The rate of convergence to Nash equilibrium of continuous fictitious play is determined for a generic set of utilities and initial beliefs in 2 x 2 games. In addition, an example is provided comparing the rate of convergence of discrete fictitious play to the rate for continuous fictitious play. Finally, the convergent dynamic of fictitious play is related to the nonconvergent gradient process dynamic in 2 x 2 games.I would like to thank Jim Jordan for many helpful discussions and for detailed comments on this paper. I also thank an anonymous referee for several helpful suggestions.     

Fictitious play in 2×n games

It is known that every discrete-time fictitious play process approaches equilibrium in nondegenerate 2×2 games, and that every continuous-time fictitious play process approaches equilibrium in nondegenerate 2×2 and 2×3 games. It has also been conjectured that convergence to the set of equilibria holds generally for nondegenerate 2×n games. We give a simple geometric proof of this for the continuous-time process, and also extend the result to discrete-time fictitious play.     

The One-Shot-Deviation Principle for Sequential Rationality

This article analyzes the fictitious play process originally proposed for strategic form games by Brown (1951) and Robinson (1951). We interpret the process as a model of preplay thinking performed by players before acting in a one-shot game. This model is one of bounded rationality. We discuss how fictitious play should then be defined for extensive form games and conclude that this is somewhat problematic. We therefore study two alternative definitions. For either of these, under a weak condition of initial uncertainty, a convergence point of a fictitious play sequence is a sequential equilibrium. For generic games of perfect information initial uncertainty also implies convergence of fictitious play.Journal of Economic LiteratureClassification Number: C72.     

Fictitious Play in Extensive Form Games

This article analyzes the fictitious play process originally proposed for strategic form games by Brown (1951) and Robinson (1951). We interpret the process as a model of preplay thinking performed by players before acting in a one-shot game. This model is one of bounded rationality. We discuss how fictitious play should then be defined for extensive form games and conclude that this is somewhat problematic. We therefore study two alternative definitions. For either of these, under a weak condition of initial uncertainty, a convergence point of a fictitious play sequence is a sequential equilibrium. For generic games of perfect information initial uncertainty also implies convergence of fictitious play.Journal of Economic LiteratureClassification Number: C72.     

Learning in games with strategic complementarities revisited

Fictitious play is a classical learning process for games, and games with strategic complementarities are an important class including many economic applications. Knowledge about convergence properties of fictitious play in this class of games is scarce, however. Beyond games with a unique equilibrium, global convergence has only been claimed for games with diminishing returns [V. Krishna, Learning in games with strategic complementarities, HBS Working Paper 92-073, Harvard University, 1992]. This result remained unpublished, and it relies on a specific tie-breaking rule. Here we prove an extension of it by showing that the ordinal version of strategic complementarities suffices. The proof does not rely on tie-breaking rules and provides some intuition for the result.     

Regret-based continuous-time dynamics

Regret-based dynamics have been introduced and studied in the context of discrete-time repeated play. Here we carry out the corresponding analysis in continuous time. We observe that, in contrast to (smooth) fictitious play or to evolutionary models, the appropriate state space for this analysis is the space of distributions on the product of the players' pure action spaces (rather than the product of their mixed action spaces). We obtain relatively simple proofs for some results known in the discrete case (related to ‘no-regret’ and correlated equilibria), and also a new result on two-person potential games (for this result we also provide a discrete-time proof).     

Fictitious Play, Shapley Polygons, and the Replicator Equation

For many normal form games, the limiting behavior of fictitious play and the time-averaged replicator dynamics coincide. In particular, we show this for three examples, where this limit is not a Nash equilibrium, but a Shapley polygon. Journal of Economic Literature Classification Numbers: C72, C73.     

On the Rate of Convergence of Continuous-Time Fictitious Play

This paper shows, first, that continuous-time fictitious play converges (in both payoff and strategy terms) uniformly at ratet ^{− 1}in any finite two-person zero-sum game. The proof is, in essence, a simple Lyapunov-function argument. The convergence of discrete-time fictitious play is a straightforward corollary of this result. The paper also shows that continuous-time fictitious play converges in all finite weighted-potential games. In this case, the convergence is not uniform. It is conjectured, however, that any given continuous-time fictitious play of a finite weighted-potential game converges (in both payoff and strategy terms) at ratet ^{− 1}.Journal of Economic LiteratureClassification Numbers: C6, C7.     

A family of supermodular Nash mechanisms implementing Lindahl allocations

Summary. We present a family of mechanisms which implement Lindahl allocations in Nash equilibrium. With quasilinear utility functions this family of mechanisms are supermodular games, which implies that they converge to Nash equilibrium under a wide class of learning dynamics. Received: April 27, 2000; revised version: January 16, 2001     

On the convergence of reinforcement learning

This paper examines the convergence of payoffs and strategies in Erev and Roth's model of reinforcement learning. When all players use this rule it eliminates iteratively dominated strategies and in two-person constant-sum games average payoffs converge to the value of the game. Strategies converge in constant-sum games with unique equilibria if they are pure or if they are mixed and the game is 2×2. The long-run behaviour of the learning rule is governed by equations related to Maynard Smith's version of the replicator dynamic. Properties of the learning rule against general opponents are also studied.     

Learning Dynamics in Games with Stochastic Perturbations

Consider a generalization of fictitious play in which agents′ choices are perturbed by incomplete information about what the other side has done, variability in their payoffs, and unexplained trembles. These perturbed best reply dynamics define a nonstationary Markov process on an infinite state space. It is shown, using results from stochastic approximation theory, that for 2 × 2 games it converges almost surely to a point that lies close to a stable Nash equilibrium, whether pure or mixed. This generalizes a result of Fudenherg and Kreps, who demonstrate convergence when the game has a unique mixed equilibrium. Journal of Economic Literature Classification Numbers: 000, 000, 000.     

Rage against the machines: how subjects play against learning algorithms

We use a large-scale internet experiment to explore how subjects learn to play against computers that are programmed to follow one of a number of standard learning algorithms. The learning theories are (unbeknown to subjects) a best response process, fictitious play, imitation, reinforcement learning, and a trial & error process. We explore how subjects’ performances depend on their opponents’ learning algorithm. Furthermore, we test whether subjects try to influence those algorithms to their advantage in a forward-looking way (strategic teaching). We find that strategic teaching occurs frequently and that all learning algorithms are subject to exploitation with the notable exception of imitation.     

Fictitious play property of the Nash demand game

We show that the Nash demand game has the fictitious play property. We also show that almost every fictitious play process and its associated belief path converge to a pure-strategy Nash equilibrium in the Nash demand game.