Learning and equilibrium as useful approximations: Accuracy of prediction on randomly selected constant sum games

There is a good deal of miscommunication among experimenters and theorists about how to evaluate a theory that can be rejected by sufficient data, but may nevertheless be a useful approximation. A standard experimental design reports whether a general theory can be rejected on an informative test case. This paper, in contrast, reports an experiment designed to meaningfully pose the question: “how good an approximation does a theory provide on average.” It focuses on a class of randomly selected games, and estimates how many pairs of experimental subjects would have to be observed playing a previously unexamined game before the mean of the experimental observations would provide a better prediction than the theory about the behavior of a new pair of subjects playing this game. We call this quantity the model’s equivalent number of observations, and explore its properties. This research was supported by a grant from the U.S. National Science Foundation and the USA–Israel Binational Science Foundation. We are very grateful for helpful conversations with David Budescu, Jerry Busemeyer, Gary Chamberlain, Paul Feigin, Dave Krantz, Jack Porter, Tom Wallsten, Wolfgang Viechtbauer, and Richard Zeckhauser.