The wisdom of crowds: The potential of online communities as a tool for data analysis

Online communities have become an important source for knowledge and new ideas. This paper considers the potential of crowdsourcing as a tool for data analysis to address the increasing problems faced by companies in trying to deal with “Big Data”. By exposing the problem to a large number of participants proficient in different analytical techniques, crowd competitions can very quickly advance the technical frontier of what is possible using a given dataset. The empirical setting of the research is Kaggle, the world?s leading online platform for data analytics, which operates as a knowledge broker between companies aiming to outsource predictive modelling competitions and a network of over 100,000 data scientists that compete to produce the best solutions. The paper follows an exploratory case study design and focuses on the efforts by Dunnhumby, the consumer insight company behind the success of the Tesco Clubcard, to find and lever the enormous potential of the collective brain to predict shopper behaviour. By adopting a crowdsourcing approach to data analysis, Dunnhumby were able to extract information from their own data that was previously unavailable to them. Significantly, crowdsourcing effectively enabled Dunnhumby to experiment with over 2000 modelling approaches to their data rather than relying on the traditional internal biases within their R&D units.