The online laboratory: conducting experiments in a real labor market

Online labor markets have great potential as platforms for conducting experiments. They provide immediate access to a large and diverse subject pool, and allow researchers to control the experimental context. Online experiments, we show, can be just as valid—both internally and externally—as laboratory and field experiments, while often requiring far less money and time to design and conduct. To demonstrate their value, we use an online labor market to replicate three classic experiments. The first finds quantitative agreement between levels of cooperation in a prisoner’s dilemma played online and in the physical laboratory. The second shows—consistent with behavior in the traditional laboratory—that online subjects respond to priming by altering their choices. The third demonstrates that when an identical decision is framed differently, individuals reverse their choice, thus replicating a famed Tversky-Kahneman result. Then we conduct a field experiment showing that workers have upward-sloping labor supply curves. Finally, we analyze the challenges to online experiments, proposing methods to cope with the unique threats to validity in an online setting, and examining the conceptual issues surrounding the external validity of online results. We conclude by presenting our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.     

The challenge of understanding decisions in experimental studies of common pool resource governance

Common pool resource experiments in the laboratory and the field have provided insights that have contrasted to those derived from conventional non-cooperative game theory. Contrary to predictions from non-cooperative game theory, participants are sometimes willing to restrain voluntarily from over extracting resources and use costly punishment to sanction other participants. Something as simple as face-to-face communication has been shown to increase average earnings significantly. In the next generation of experiments, both in the laboratory and in the field, we need to extract more information that provides insight concerning why people make the decisions they make. More information is needed concerning attributes of individuals as well as the social and social-ecological context in which they interact that may give rise to such deviations from theoretical predictions. In the process of extracting more information from participants and the contexts in which they interact, we face several methodological and ethical challenges which we address in this paper.     

Optimal design of experiments to identify latent behavioral types

Bayesian optimal experiments that maximize the information gained from collected data are critical to efficiently identify behavioral models. We extend a seminal method for designing Bayesian optimal experiments by introducing two computational improvements that make the procedure tractable: (1) a search algorithm from artificial intelligence that efficiently explores the space of possible design parameters, and (2) a sampling procedure which evaluates each design parameter combination more efficiently. We apply our procedure to a game of imperfect information to evaluate and quantify the computational improvements. We then collect data across five different experimental designs to compare the ability of the optimal experimental design to discriminate among competing behavioral models against the experimental designs chosen by a “wisdom of experts” prediction experiment. We find that data from the experiment suggested by the optimal design approach requires significantly less data to distinguish behavioral models (i.e., test hypotheses) than data from the experiment suggested by experts. Substantively, we find that reinforcement learning best explains human decision-making in the imperfect information game and that behavior is not adequately described by the Bayesian Nash equilibrium. Our procedure is general and computationally efficient and can be applied to dynamically optimize online experiments.

     

The limits of self-governance when cooperators get punished: Experimental evidence from urban and rural Russia

We report evidence from public goods experiments with and without punishment which we conducted in Russia with 566 urban and rural participants of young and mature age cohorts. Russia is interesting for studying voluntary cooperation because of its long history of collectivism, and a huge urban–rural gap. In contrast to previous experiments we find no cooperation-enhancing effect of punishment. An important reason is that there is punishment of contributors in all four subject pools. Thus, punishment can also undermine the scope for self-governance in the sense of high levels of voluntary cooperation that are sustained by sanctioning free riders only.     

A comparative statics analysis of punishment in public-good experiments

This paper provides a comparative-statics analysis of punishment in public-good experiments. We vary the effectiveness of punishment, that is, the factor by which punishment reduces the punished player’s income. The data show that contributions increase monotonically in punishment effectiveness. High effectiveness leads to near complete cooperation and welfare improvements. Below a certain threshold, however, punishment cannot prevent the decay of cooperation. In these cases, punishment opportunities reduce welfare. The results suggest that the experimenter’s choice of the punishment effectiveness is of great importance for the experimental outcome.

---

Electronic Supplementary Material  The online version of this article () contains supplementary material, which is available to authorized users.      

On punishment and well-being

The existence of punishment opportunities has been shown to cause efficiency in some public goods experiments to increase considerably. In this paper we ask whether punishment also has a downside in terms of process dissatisfaction. We conduct an experiment to study the conjecture that an environment with strong punishment possibilities may lead to higher material payoffs but lower subjective well-being, in comparison with weaker punishment or no punishment possibilities at all. The more general motivation for our study stems from the notion that people’s subjective well-being may be affected by the institutional environment they find themselves in. Our findings show that harsher punishment possibilities lead to significantly higher well-being, controlling for earnings and other relevant variables. These results complement the evidence on the neural basis of altruistic punishment reported in De Quervain et al. (2004).     

Virtual world experimentation: An exploratory study

We explore the scientific potential of virtual worlds for experimental economics in terms of the subject pools and experimental platforms they present. Our results offer tentative, qualified support for virtual world experimentation. Overall, the behaviour of virtual subjects recruited, incentivised and observed within Second Life across a range of five standard experimental games was not found to differ significantly from established standard results. In addition, we identify certain methodological opportunities and challenges which confront virtual world experimenters.     

Experiments,policy, and theory in development economics: a response to Glenn Harrison’s ‘field experiments and methodological intolerance’

In ‘Field Experiments and Methodological Intolerance,’ Glenn Harrison develops a criticism of randomized field experiments and denounces ‘intolerance’ for lab experiments and a ‘disconnect from theory.’ We argue that (i) lab experiments and RCTs are based on different methodological approaches and therefore fulfill different scientific objectives; (ii) key features of the RCT methodology make it highly relevant for policy-making, which accounts for a large part of its uptake; (iii) RCTs foster a convergence of interests between policy-makers and researchers around the study of economic mechanisms, which makes it possible to challenge and improve economic theory based on real-world facts; and (iv) hybrid forms of experimentation are being increasingly used and open promising avenues. In sum, economic research based on RCTs is more and more connected to theory and other forms of experiment, while also helping policy-makers tackle daunting problems of poverty and exclusion.     

Experimenting and learning with localized direct communication

We report an experiment in which subjects may learn from each other. Specifically, a “queue” of players who are identically informed ex ante make decisions in sequence over two lotteries. Every player except the first in the queue observes (only) his immediate predecessor’s choice and payoff before making his own decision. In equilibrium decisions are identical from the first or second player onwards in all experimental conditions. However, complete adherence to equilibrium play is seldom observed in our experiment. We further analyze our data using a quantal response equilibrium approach and test for behavioral regularities related to base rate fallacy/conservatism bias, social conformity/rebelliousness, and preference for experimentation (preferring the lottery with potentially more information spillover value). Our estimations reveal a consistent preference for experimentation across conditions, and further analysis offers some support to our surmise that this behavioral regularity is due, in part, to an attempt to influence others behind in the queue.     

Design and evaluation of an economic experiment via the Internet

The paper investigates whether Internet experiments are an appropriate alternative to traditional laboratory experiments. For an economic experiment of individual decision making results obtained over the Internet are compared to results obtained in the laboratory using exactly the same software. Of particular interest are differences in individual behavior. Our main findings are: (1) running our own experiment on the Internet and in the laboratory generated similar data when economic decision behavior is concerned. (2) Variance in economic decision behavior is generally higher on the Internet experiment. (3) Decision times are shorter on the Internet. (4) Internet software provides a helpful platform for implementing economic experiments. The paper also reports on design challenges and how we have solved them.     

Mixing the carrots with the sticks: third party punishment and reward

While the opportunity to punish selfish and reward generous behavior coexist in many instances in daily life, in most laboratory studies, the demand for punishment and reward are studied separately from one another. This paper presents the results from an experiment measuring the demand for reward and punishment by ‘unaffected’ third parties, separately and jointly. We find that the demand for costly punishment is substantially lower when individuals are also given the ability to reward. Similarly, the demand for costly reward is lower when individuals can also punish. The evidence indicates the reason for this is that costly punishment and reward are not only used to alter the material payoff of others as assumed by recent economic models, but also as a signal of disapproval and approval of others’ actions, respectively. When the opportunity exists, subjects often choose to withhold reward as a form of costless punishment, and to withhold punishment as a form of costless reward. We conclude that restricting the available options to punishing (rewarding) only, may lead to an increase in the demand for costly punishment (reward).     

Punishment in a Regulatory Setting: Experimental Evidence from the VCM

To analyze the effectiveness of punishment in inducing regulatory compliance, we modify a standard public goods experiment to include a financial penalty for free riding. The design allows us to vary both punishment probability and severity. We introduce the punishment mechanism in both a one-time and a repeated treatment and find that compliance (contributing to the public good) is increasing in expected punishment cost in both treatments. We also find that punishment severity has a larger effect on behavior than punishment probability. In the repeated treatment, we find that past punishment has a negative rather than positive effect on compliance.     

On the acceptance of apologies

An apology is a strong and cheap device to restore social or economic relationships that have been disturbed. In a laboratory experiment in which apologies emerge endogenously, we find that harmdoers use apologies in particular if they fear punishment and if their intentions cannot be easily inferred. After offenses with ambiguous intention punishment for apologizers is lower than for non-apologizers. Victims expect an apology and punish if they do not receive one. An apology does not help at all after clearly intentionally committed offenses. On the contrary, after such offenses an apology strongly increases punishment compared to remaining silent.     

Predatory pricing in a multiple market experiment: A note

Isaac and Smith (1985a,b) present a strong behavioral case that predatory pricing is not an observable laboratory phenomenon. We propose several modifications of their design, resulting in a multiple-market experiment in which predatory pricing is present. The primary modification involves conducting several markets concurrently, providing an active ‘escape’ opportunity for any prey and a potential reward to a reputation for predation. These results provide a ‘behavioral existence proof’ for the predatory pricing phenomenon. They do not address the issue of the phenomenon's behavioral robustness.     

Holding back from theory: limits and methodological alternatives of randomized field experiments in development economics

ABSTRACT

In this paper, we critically and constructively examine the methodology of evidence-based development economics, which deploys randomized field experiments (RFEs) as its main tool. We describe the context in which this movement started, and illustrate in detail how RFEs are designed and implemented in practice, drawing on a series of experiments by Pascaline Dupas and her colleagues on the use of bednets, saving and governance in Kenya. We show that this line of experiments have evolved to address the limitation of obtaining policy-relevant insights from RFEs alone, characterized as their lack of external validity in the literature. After examining the two prominent responses by leading figures of evidence-based development economics, namely machine learning and structured speculation, we propose an alternative methodological strategy that incorporates two sub-fields, namely experimental economics and behavioral economics, to complement RFEs in investigating the data-generating process underlying the treatment effects of RFEs. This strategy highlights promising methodological developments in RFEs neither captured by the two proposals nor recognized by methodologists, and also guides how to combine different sub-fields of economics.     

Imitation—theory and experimental evidence

We introduce a generalized theoretical approach to study imitation and subject it to rigorous experimental testing. In our theoretical analysis we find that the different predictions of previous imitation models are mainly explained by different informational assumptions, and to a lesser extent by different behavioral rules. In a laboratory experiment we test the different theories by systematically varying information conditions. We find significant effects of seemingly innocent changes in information. Moreover, the generalized imitation model predicts the differences between treatments well. The data provide support for imitation on the individual level, both in terms of choice and in terms of perception. Furthermore, individuals’ propensity to imitate more successful actions is increasing in payoff differences.     

Field experiments in economics: The past, the present, and the future

This study presents an overview of modern field experiments and their usage in economics. Our discussion focuses on three distinct periods of field experimentation that have influenced the economics literature. The first might well be thought of as the dawn of “field” experimentation: the work of Neyman and Fisher, who laid the experimental foundation in the 1920s and 1930s by conceptualizing randomization as an instrument to achieve identification via experimentation with agricultural plots. The second, the large-scale social experiments conducted by government agencies in the mid-twentieth century, moved the exploration from plots of land to groups of individuals. More recently, the nature and range of field experiments has expanded, with a diverse set of controlled experiments being completed outside of the typical laboratory environment. With this growth, the number and types of questions that can be explored using field experiments has grown tremendously. After discussing these three distinct phases, we speculate on the future of field experimental methods, a future that we envision including a strong collaborative effort with outside parties, most importantly private entities.     

Policy making and rent-dissipation: An experimental test

We present a transfer-seeking model of political economy that links the theory of Becker (1983) with Tullock-type models of politically contestable rents. In our model the size of the transfer is determined endogenously, and over-dissipation of rents is predicted even under conditions of risk-neutrality and perfect rationality. We implement an empirical test of this model by collecting behavioral data in a laboratory experiment. We confirm the existence of behavior that leads to over-dissipation of rents in games with both symmetric and asymmetric political power. To the extent that the transfer-seeking costs are social costs, our findings imply that the total costs of running government might be greatly underestimated if the value of the rent is used as a proxy for the rent-seeking cost. We also confirm the hypotheses that lowering the political power of one player can lead to smaller rent-seeking expenditures and to larger transfers Electronic Supplementary Material Supplementary material is available in the online version of this article at . JEL Classification C91, D72     

The BCD of response time analysis in experimental economics

For decisions in the wild, time is of the essence. Available decision time is often cut short through natural or artificial constraints, or is impinged upon by the opportunity cost of time. Experimental economists have only recently begun to conduct experiments with time constraints and to analyze response time (RT) data, in contrast to experimental psychologists. RT analysis has proven valuable for the identification of individual and strategic decision processes including identification of social preferences in the latter case, model comparison/selection, and the investigation of heuristics that combine speed and performance by exploiting environmental regularities. Here we focus on the benefits, challenges, and desiderata of RT analysis in strategic decision making. We argue that unlocking the potential of RT analysis requires the adoption of process-based models instead of outcome-based models, and discuss how RT in the wild can be captured by time-constrained experiments in the lab. We conclude that RT analysis holds considerable potential for experimental economics, deserves greater attention as a methodological tool, and promises important insights on strategic decision making in naturally occurring environments.     

The evolution of sanctioning institutions: an experimental approach to the social contract

A vast amount of empirical and theoretical research on public good games indicates that the threat of punishment can curb free-riding in human groups engaged in joint enterprises. Since punishment is often costly, however, this raises an issue of second-order free-riding: indeed, the sanctioning system itself is a common good which can be exploited. Most investigations, so far, considered peer punishment: players could impose fines on those who exploited them, at a cost to themselves. Only a minority considered so-called pool punishment. In this scenario, players contribute to a punishment pool before engaging in the joint enterprise, and without knowing who the free-riders will be. Theoretical investigations (Sigmund et al., Nature 466:861–863, 2010) have shown that peer punishment is more efficient, but pool punishment more stable. Social learning, i.e., the preferential imitation of successful strategies, should lead to pool punishment if sanctions are also imposed on second-order free-riders, but to peer punishment if they are not. Here we describe an economic experiment (the Mutual Aid game) which tests this prediction. We find that pool punishment only emerges if second-order free riders are punished, but that peer punishment is more stable than expected. Basically, our experiment shows that social learning can lead to a spontaneously emerging social contract, based on a sanctioning institution to overcome the free rider problem.