Testing the TASP: An experimental investigation of learning in games with unstable equilibria

We report experiments studying mixed strategy Nash equilibria that are theoretically stable or unstable under learning. The Time Average Shapley Polygon (TASP) predicts behavior in the unstable case. We study two versions of Rock-Paper-Scissors that include a fourth strategy, Dumb. The unique Nash equilibrium is identical in the two games, but the predicted frequency of Dumb is much higher in the game where the NE is stable. Consistent with TASP, the observed frequency of Dumb is lower and play is further from Nash in the high payoff unstable treatment. However, Dumb is played too frequently in all treatments.     

Fictitious Play, Shapley Polygons, and the Replicator Equation

For many normal form games, the limiting behavior of fictitious play and the time-averaged replicator dynamics coincide. In particular, we show this for three examples, where this limit is not a Nash equilibrium, but a Shapley polygon. Journal of Economic Literature Classification Numbers: C72, C73.     

Evolution in games with randomly disturbed payoffs

We consider a simple model of stochastic evolution in population games. In our model, each agent occasionally receives opportunities to update his choice of strategy. When such an opportunity arises, the agent selects a strategy that is currently optimal, but only after his payoffs have been randomly perturbed. We prove that the resulting evolutionary process converges to approximate Nash equilibrium in both the medium run and the long run in three general classes of population games: stable games, potential games, and supermodular games. We conclude by contrasting the evolutionary process studied here with stochastic fictitious play.     

The geometry of inductive reasoning in games

Summary.  This paper contributes to the recent focus on dynamics in noncooperative games when players use inductive learning. The most well-known inductive learning rule, Brown’s fictitious play, is known to converge for games, yet many examples exist where fictitious play reasoning fails to converge to a Nash equilibrium. Building on ideas from chaotic dynamics, this paper develops a geometric conceptualization of instability in games, allowing for a reinterpretation of existing results and suggesting avenues for new results. Received: October 27, 1995 revised version May 2, 1996     

Fictitious play property of the Nash demand game

We show that the Nash demand game has the fictitious play property. We also show that almost every fictitious play process and its associated belief path converge to a pure-strategy Nash equilibrium in the Nash demand game.     

Fictitious play in 2×n games

It is known that every discrete-time fictitious play process approaches equilibrium in nondegenerate 2×2 games, and that every continuous-time fictitious play process approaches equilibrium in nondegenerate 2×2 and 2×3 games. It has also been conjectured that convergence to the set of equilibria holds generally for nondegenerate 2×n games. We give a simple geometric proof of this for the continuous-time process, and also extend the result to discrete-time fictitious play.     

Generalised weakened fictitious play

A general class of adaptive processes in games is developed, which significantly generalises weakened fictitious play [Van der Genugten, B., 2000. A weakened form of fictitious play in two-person zero-sum games. Int. Game Theory Rev. 2, 307–328] and includes several interesting fictitious-play-like processes as special cases. The general model is rigorously analysed using the best response differential inclusion, and shown to converge in games with the fictitious play property. Furthermore, a new actor–critic process is introduced, in which the only information given to a player is the reward received as a result of selecting an action—a player need not even know they are playing a game. It is shown that this results in a generalised weakened fictitious play process, and can therefore be considered as a first step towards explaining how players might learn to play Nash equilibrium strategies without having any knowledge of the game, or even that they are playing a game.     

The rate of convergence of continuous fictitious play★

Summary The rate of convergence to Nash equilibrium of continuous fictitious play is determined for a generic set of utilities and initial beliefs in 2 x 2 games. In addition, an example is provided comparing the rate of convergence of discrete fictitious play to the rate for continuous fictitious play. Finally, the convergent dynamic of fictitious play is related to the nonconvergent gradient process dynamic in 2 x 2 games.I would like to thank Jim Jordan for many helpful discussions and for detailed comments on this paper. I also thank an anonymous referee for several helpful suggestions.     

Regret-based continuous-time dynamics

Regret-based dynamics have been introduced and studied in the context of discrete-time repeated play. Here we carry out the corresponding analysis in continuous time. We observe that, in contrast to (smooth) fictitious play or to evolutionary models, the appropriate state space for this analysis is the space of distributions on the product of the players' pure action spaces (rather than the product of their mixed action spaces). We obtain relatively simple proofs for some results known in the discrete case (related to ‘no-regret’ and correlated equilibria), and also a new result on two-person potential games (for this result we also provide a discrete-time proof).     

Learning Dynamics in Games with Stochastic Perturbations

Consider a generalization of fictitious play in which agents′ choices are perturbed by incomplete information about what the other side has done, variability in their payoffs, and unexplained trembles. These perturbed best reply dynamics define a nonstationary Markov process on an infinite state space. It is shown, using results from stochastic approximation theory, that for 2 × 2 games it converges almost surely to a point that lies close to a stable Nash equilibrium, whether pure or mixed. This generalizes a result of Fudenherg and Kreps, who demonstrate convergence when the game has a unique mixed equilibrium. Journal of Economic Literature Classification Numbers: 000, 000, 000.     

Self-tuning experience weighted attraction learning in games

Self-tuning experience weighted attraction (EWA) is a one-parameter theory of learning in games. It addresses a criticism that an earlier model (EWA) has too many parameters, by fixing some parameters at plausible values and replacing others with functions of experience so that they no longer need to be estimated. Consequently, it is econometrically simpler than the popular weighted fictitious play and reinforcement learning models. The functions of experience which replace free parameters “self-tune” over time, adjusting in a way that selects a sensible learning rule to capture subjects’ choice dynamics. For instance, the self-tuning EWA model can turn from a weighted fictitious play into an averaging reinforcement learning as subjects equilibrate and learn to ignore inferior foregone payoffs. The theory was tested on seven different games, and compared to the earlier parametric EWA model and a one-parameter stochastic equilibrium theory (QRE). Self-tuning EWA does as well as EWA in predicting behavior in new games, even though it has fewer parameters, and fits reliably better than the QRE equilibrium benchmark.     

The supercore for normal-form games

This paper analyzes the supercore of a system derived from a normal-form game. For the case of a finite game with pure strategies, we define a sequence of games and show that the supercore coincides with the set of Nash equilibria of the last game in that sequence. This result is illustrated with the characterization of the supercore for the n-person prisoner's dilemma. With regard to the mixed extension of a normal-form game, we show that the set of Nash equilibrium profiles coincides with the supercore for games with a finite number of Nash equilibria.     

Mixed equilibria are unstable in games of strategic complements

In games with strict strategic complementarities, properly mixed Nash equilibria—equilibria that are not in pure strategies—are unstable for a broad class of learning dynamics.     

On the impossibility of achieving no regrets in repeated games

Regret-minimizing strategies for repeated games have been receiving increasing attention in the literature. These are simple adaptive behavior rules that lead to no regrets and, if followed by all players, exhibit nice convergence properties: the average play converges to correlated equilibrium, or even to Nash equilibrium in certain classes of games. However, the no-regret property relies on a strong assumption that each player treats her opponents as unresponsive and fully ignores the opponents’ possible reactions to her actions. We show that if at least one player is slightly responsive, it is impossible to achieve no regrets, and convergence results for regret minimization with responsive opponents are unknown.     

Competitive behavior in market games: Evidence and theory

We explore whether competitive outcomes arise in an experimental implementation of a market game, introduced by Shubik (1973) [21]. Market games obtain Pareto inferior (strict) Nash equilibria, in which some or possibly all markets are closed. We find that subjects do not coordinate on autarkic Nash equilibria, but favor more efficient Nash equilibria in which all markets are open. As the number of subjects participating in the market game increases, the Nash equilibrium they achieve approximates the associated competitive equilibrium of the underlying economy. Motivated by these findings, we provide a theoretical argument for why evolutionary forces can lead to competitive outcomes in market games.     

Brown's original fictitious play

What modern game theorists describe as “fictitious play” is not the learning process George W. Brown defined in his 1951 paper. Brown's original version differs in a subtle detail, namely the order of belief updating. In this note we revive Brown's original fictitious play process and demonstrate that this seemingly innocent detail allows for an extremely simple and intuitive proof of convergence in an interesting and large class of games: nondegenerate ordinal potential games.     

Coordination and learning in games with strategic substitutes and complements

This paper experimentally compares the impact of the presence of strategic substitutes (GSS) and complements (GSC) on players’ ability to successfully play equilibrium strategies. By exploiting a simple property of the ordering on strategy spaces, our design allows us to isolate these effects by avoiding other confounding factors that are present in more complex settings, such as market games. We find that the presence of strategic complementarities significantly improves the rate of Nash play, but that this effect is driven mainly by early rounds of play. This suggests that GSS may be more difficult to learn initially, but that given sufficient time, the theoretically supported globally stable equilibrium offers a good prediction in both settings. We also show that increasing the degree of substitutability or complementarity does not significantly improve the rate of Nash play in either setting, which builds on the findings of previous studies.     

Decompositions and potentials for normal form games

A basic model of commitment is to convert a two-player game in strategic form to a “leadership game” with the same payoffs, where one player, the leader, commits to a strategy, to which the second player always chooses a best reply. This paper studies such leadership games for games with convex strategy sets. We apply them to mixed extensions of finite games, which we analyze completely, including nongeneric games. The main result is that leadership is advantageous in the sense that, as a set, the leader's payoffs in equilibrium are at least as high as his Nash and correlated equilibrium payoffs in the simultaneous game. We also consider leadership games with three or more players, where most conclusions no longer hold.     

An experimental study of information and mixed-strategy play in the three-person matching-pennies game

Summary. Recent experiments on mixed-strategy play in experimental games reject the hypothesis that subjects play a mixed strategy even when that strategy is the unique Nash equilibrium prediction. However, in a three-person matching-pennies game played with perfect monitoring and complete payoff information, we cannot reject the hypothesis that subjects play the mixed-strategy Nash equilibrium. Given this support for mixed-strategy play, we then consider two qualitatively different learning theories (sophisticated Bayesian and naive Bayesian) which predict that the amount of information given to subjects will determine whether they can learn to play the predicted mixed strategies. We reject the hypothesis that subjects play the symmetric mixed-strategy Nash equilibrium when they do not have complete payoff information. This finding suggests that players did not use sophisticated Bayesian learning to reach the mixed-strategy Nash equilibrium. Received: August 9, 1996; revised version: October 21, 1998     

Fictitious Play in Extensive Form Games

This article analyzes the fictitious play process originally proposed for strategic form games by Brown (1951) and Robinson (1951). We interpret the process as a model of preplay thinking performed by players before acting in a one-shot game. This model is one of bounded rationality. We discuss how fictitious play should then be defined for extensive form games and conclude that this is somewhat problematic. We therefore study two alternative definitions. For either of these, under a weak condition of initial uncertainty, a convergence point of a fictitious play sequence is a sequential equilibrium. For generic games of perfect information initial uncertainty also implies convergence of fictitious play.Journal of Economic LiteratureClassification Number: C72.