How Cloudy a Crystal Ball: A Psychometric Assessment of Concept Testing*

Although traditional and conjoint forms of concept testing play an important role in the new product development process, they largely ignore data quality issues, as evidenced by the traditional reliance on the percent Top‐2‐Box scores heuristic. The purpose of this research is to reconsider the design of concept testing from a measurement theory (generalizability theory) perspective and to use it to suggest some ways to improve the psychometric quality of concept testing. Generalizability theory is employed because it can account for the multiple facets of variation in concept testing, and it enables a concept test to be designed to provide a required level of accuracy for decision making in the most effective way, whether the purpose of measurement is to scale concepts or something else, such as to scale respondents. The paper identifies four types of sources—concept‐related factors, response task factors, situational factors, and respondent factors—that can contribute to the observed variation in concept testing and develops six research propositions that summarize what is known or assumed about their contribution to observed score variance. Four secondary data sets from different concept testing contexts are then used to test the propositions. The results provide new insights into the design of concept tests and the psychometric quality of the concept testing data: (1) the concepts facet is not a major contributor to response variation; (2) of the response task factors, concept formulations are a trivial source of variance, but items are not always a trivial source of variance; (3) the situational factors that are investigated are trivial sources of variance; (4) respondents are always a major contributor to the total variation; (5) concepts by respondents are not always a major contributor and the other interactions are often not trivial; and (6) residual error is always a major source of variance. Additionally, the analyses of the secondary data sets enable some useful managerial conclusions to be drawn about the design of concept testing. First, the sample size needed to reliably scale concepts depends on the types of concepts being tested. Second, averaging over items provides considerably more reliable information than relying on a single item. Third, which specific item performs best is inconsistent and very context specific. The popular purchase intention item is never the best single item to use. Fourth, not much is gained by sampling levels of the response task factors. Finally, concept testing should be designed to meet the needs of specific managerial tasks.     

The Predictive Power of Internet-Based Product Concept Testing Using Visual Depiction and Animation

One critical step in new product development is selecting from among multiple possible product concepts the one that the firm will carry forward into the marketplace. There is a need for low‐cost, parallel testing of the appeal of new product concepts, the results of which closely mirror ultimate market performance. In this article, the authors first describe an Internet‐based product concept testing method they developed that incorporates virtual prototypes of new product concepts, substituting them for physical prototypes. The method can be used with either static representations of the products or with dynamic representations that demonstrate how the product works through a simulated video clip of its operation. The objective of this method is to allow design teams to select the best of several new concepts within a product category with which to proceed, without having to develop physical prototypes. The authors then provide a rigorous test of both virtual prototype methods against tests using both physical prototypes and attribute‐only (i.e., no visuals), full‐profile conjoint analysis. Nine concepts compete against two actual products in the tests. Market shares from the test using the physical prototypes are defined as the “actual” market shares. Predicted market shares for the attribute‐only, full‐profile conjoint analysis and each of the two virtual prototype methods are compared to those obtained for the physical prototypes. Both static and animated virtual prototype tests produced market shares that closely mirrored those obtained with the physical products, outperforming the set of predictions across the full range of products produced in the attribute‐only conjoint analysis. Interestingly, the attribute‐only conjoint analysis identified the top three products, in correct order. It was unable to differentiate performance below these top three products. Furthermore, it predicted market shares for the top three products to be well below those achieved using physical prototypes. As virtual prototypes cost considerably less to build and test than their physical counterparts, design teams using Internet‐based product concept research may be able to afford to explore a much larger number of concepts. Virtual prototypes and the testing methods associated with them may help reduce the uncertainty and cost of new product introductions by allowing more ideas to be concept tested in parallel with target consumers.     

Enhancing Concept Test Validity by Using Expert Consumers

In standard concept testing practice, consumers may be invited to participate in a test if they use or possess the product. However, merely using or possessing a product is no guarantee that a consumer has the level of product knowledge that is necessary for judging the concept. Conducting a concept test with consumers who lack the necessary product knowledge may jeopardize the validity of the test results. That is, the results of such a concept test may not accurately indicate how consumers will evaluate the real product. To ensure the validity of concept test results, Jan Schoormans, Roland Ortt, and Cees de Bont suggest that consumers who are invited to participate in a concept test should possess a degree of product knowledge. When a consumer is asked to evaluate a concept, their product expertise allows them to understand product information faster, fill in missing information, and learn more easily. Consumers with product expertise are better able to discriminate between important and unimportant aspects of a product. They are also better able to infer benefits from a product's physical attributes. To explore the effects of consumer expertise on the quality of the evaluations provided by concept tests, the authors conducted two experiments, both of which resemble actual concept tests. The first experiment examines the effect of consumer expertise on the results of a concept test for a major innovation, Videotext. This experiment tests the hypothesis that the similarity between the evaluations of a concept and an actual product will be greater for consumers with a high level of product-category expertise than for consumers with low product-category expertise. The results of the experiment clearly support the idea that product-category expertise enhances a respondent's ability to evaluate concepts in a test of major innovations. From this, it is concluded that only respondents with high product-category expertise should be used for concept tests of major innovations. The second experiment explores the effects of product expertise on consumers' evaluations of a minor innovation, a redesigned coffee maker. This experiment tests two hypotheses. First, it is proposed that consumers with high product expertise give more consistent evaluations in a concept test than consumers with low product expertise. Second, it is suggested that consumers with product expertise generate more stable evaluations over time than consumers without product expertise. The results of this experiment clearly indicate that using consumers with moderate to high levels of product expertise is beneficial to the validity of the results from concept tests of minor innovations.     

The Moderating Roles of Prior Experience and Behavioral Importance in the Predictive Validity of New Product Concept Testing

Concept testing has long been recognized as an important new product development (NPD) activity. As one of the widely used concept testing techniques, the method of intentions surveys relies on the purchase intentions of the potential buyers of new products and helps firms assess the viabilities of their new products before making major financial and nonfinancial commitments to their development. Despite the importance of intentions‐based new product concept testing and its widespread use by firms, the correspondence between initial behavioral intentions and subsequent purchase behaviors has been relatively low and heterogeneous, making it very difficult for firms to draw any useful conclusions from intentions surveys. Focusing on the predictive validity of intentions‐based new product concept testing and addressing several calls for future research to identify specific conditions making it more effective, this paper tests the moderating roles of prior experience and behavioral importance in the predictive validity of intentions‐based new product concept testing. It also tests whether people who state a positive intention and people who state a negative intention are equally accurate in their intentions. Finally, it tests the relative moderating roles of prior experience and behavioral importance in the intentions–behavior relationship. The results based on two longitudinal surveys first suggested that people's prior experience moderates the relationship between behavioral intentions and actual behaviors in a way that the relationship is stronger when prior experience is high as opposed to when it is low. Second, they showed that the level of importance that people attach to a behavior also moderates the relationship between behavioral intentions and actual behaviors such that the relationship is stronger when behavioral importance is high as opposed to when it is low. Third, they indicated that the behavioral intentions of people who state that they will not perform a behavior are more accurate than are those of people who state that they will perform it. Finally, the results suggested that the impact of behavioral importance is greater than that of prior experience. This study offers several implications. Most notably, the results can help firms better understand different factors affecting the predictive validity of intentions‐based new product concept testing and hence make more accurate new product decisions.     

Whose Crystal Ball to Choose? Individual Difference in the Generalizability of Concept Testing

The product development literature has identified several individual characteristics that could influence how subjects respond to new products in concept tests. Few of these characteristics have been thoroughly investigated. The purpose of this research is to examine whether a number of personality traits (1) do influence concept evaluation scores and (2) can be used to identify respondents who provide substantially higher‐quality data in concept testing and whether the answers to these questions change for major versus minor innovations. The data quality of the concept testing data is defined using the generalizability theory, which provides a decision‐specific G‐coefficient. Higher quality means a G‐coefficient closer to 1 for a particular managerial decision. A Web‐based study to concept test 10 appliance innovations on multiple occasions was conducted among 105 panelists from the Institute for Online Consumer Studies (IOCS). During the concept testing, respondents' innovativeness, change‐seeking tendency, and propensity to exert cognitive effort were also measured. The results showed that the respondent characteristics influence the mean evaluation of the concepts and the psychometric quality of the concept testing data: (1) there is a significant linear relationship between concept scores and all of the innovativeness scales and change‐seeking measures; (2) the effect of innovativeness on concept testing outcomes is even more substantial for major innovations than for minor innovations; (3) the study provides evidence that the quality of concept testing data provided by respondents varies substantially with their innovativeness, whereas the differences are more modest when scaling just minor innovations; (5) there are also strong effects on data quality for the Need to Evaluate scale used to capture cognitive effort characteristics; and (6) there is little effect of segmenting on social desirability on data quality. Managerially, the current results indicate that a product manager wanting to concept test a pool of appliance concepts can benefit from screening for the respondents who will provide higher‐quality concept testing data. For example, respondents who are high on domain‐specific innovativeness provide the highest‐quality concept testing data for both minor and major innovations. The effects of traits are stronger for major innovations, supporting the claim that subject selection is a more critical issue in concept testing of major innovations. Product managers can improve the quality of their concept testing data without an increase in cost by screening the subjects they use in concept testing.     

The virtual customer

Communication and information technologies are adding new capabilities for rapid and inexpensive customer input to all stages of the product development (PD) process. In this article we review six web‐based methods of customer input as examples of the improved Internet capabilities of communication, conceptualization, and computation. For each method we give examples of user‐interfaces, initial applications, and validity tests. We critique the applicability of the methods for use in the various stages of PD and discuss how they complement existing methods. For example, during the fuzzy front end of PD the information pump enables customers to interact with each other in a web‐based game that provides incentives for truth‐telling and thinking hard, thus providing new ways for customers to verbalize the product features that are important to them. Fast polyhedral adaptive conjoint estimation enables PD teams to screen larger numbers of product features inexpensively to identify and measure the importance of the most promising features for further development. Meanwhile, interactive web‐based conjoint analysis interfaces are moving this proven set of methods to the web while exploiting new capabilities to present products, features, product use, and marketing elements in streaming multimedia representations. User design exploits the interactivity of the web to enable users to design their own virtual products thus enabling the PD team to understand complex feature interactions and enabling customers to learn their own preferences for new products. These methods can be valuable for identifying opportunities, improving the design and engineering of products, and testing ideas and concepts much earlier in the process when less time and money is at risk. As products move toward pretesting and testing, virtual concept testing on the web enables PD teams to test concepts without actually building the product. Further, by combining virtual concepts and the ability of customers to interact with one another in a stock‐market‐like game, securities trading of concepts provides a novel way to identify winning concepts. Prototypes of all six methods are available and have been tested with real products and real customers. These tests demonstrate reliability for web‐based conjoint analysis, polyhedral methods, virtual concept testing, and stock‐market‐like trading; external validity for web‐based conjoint analysis and polyhedral methods; and consistency for web‐based conjoint analysis versus user design. We report on these tests, commercial applications, and other evaluations. © 2002 Elsevier Science Inc. All rights reserved.     

Narrative Transportation in Concept Tests for Really New Products: The Moderating Effect of Reader–Protagonist Similarity

The authors investigate the benefits of using a narrative (i.e., a storyline featuring a protagonist) to convey product information in the evaluation of really new product concepts by consumers. In the context of early product evaluation, the imagination of consumers can be guided by a narrative about a protagonist who uses the new product in a series of actions and events. In this way, a narrative can present information about the new product concept in a way that is evocative and relevant. The authors build on narratives research and study the implications of different protagonist focal characters. Further, the role of a protagonist focal character in facilitating consumer evaluations is examined, and evaluation formats (narrative versus attribute/benefit listings) are compared. Utilizing three empirical studies, this research looks at the potential effects of protagonist (dis)similarity with the reader on transportation and new product evaluation both in narrative and bulleted list evaluation formats. Study 1 shows an interactive effect of reader–protagonist similarity and evaluation format on transportation and product evaluation. The results from this study show that reader–protagonist similarity is needed for a narrative to be effective. Studies 2 and 3 provide further understanding of the effects of reader–protagonist (dis)similarity. Study 2 shows that the negative impact of a dissimilar protagonist can be mitigated by explicitly instructing the readers to imagine themselves as the protagonist, thus enabling them to fully experience the storyline. Study 3 decomposes the reader–protagonist dissimilarity and shows that not all protagonists dissimilar to the reader deliver a negative outcome. A dissimilar protagonist that is not from a dissociative out‐group for the reader effectuates a positive result. Finally, the underlying process for the observed effects is demonstrated: narrative transportation is shown to mediate the observed effects in all three studies. With these studies, the authors advance narrative transportation and social identity theory. Furthermore, the research provides practical guidelines for how narratives should be constructed and utilized to obtain consumer evaluations of product concepts in the new product development process.     

Introducing Interface Management in New Product Family Development

Creating product platforms from which many products can easily be leveraged is an issue of increasing concern for many companies. This article introduces and explores the concept of interface management (IM) in new product platform development. IM is the distinct process of developing and defining the physical platform interfaces. The concept of IM is empirically explored in two product family development projects in the Swedish manufacturing industry that have been longitudinally studied for more than 3 years. It is proposed that firms which have a product family development approach that is associated with an extensive IM process enjoy a high degree of freedom in deciding how to balance its time to market for individual products with the beneficial utilization of design familiarities across all products. Moreover, if product managers understand and explicitly focus on the IM process, the often challenging shift from a single product development approach to a product family development approach is likely to be facilitated. © 1999 Elsevier Science Inc.     

Working with concepts in the fuzzy front end: exploring the context for innovation for different types of concepts at Volvo Cars

Automotive firms are balancing the increasing needs for cost and time efficiency with the necessity of developing more innovative products to stand out on in a competitive market. The strive for efficiency has led to an increasingly structured development process with limited allowances for deviations. Previous academic work has pointed out the importance and embedded potential of the fuzzy front end, where new concepts still have the possibility to impact the new product development (NPD) process. However, most research has focused on the transfer of new technologies, while concepts based on e.g. customer or market knowledge have been more or less neglected. This paper discusses the need for alternative and contingent approaches in the front end of NPD to also consider the transfer of other types of concepts. More specifically, it addresses the need to distinguish between different types of concepts and to explore their different prerequisites in NPD. It is argued that customer- and market-based concepts experience certain difficulties due to the history and power of technology in research and development (R&D) domains in the automotive context as well as a lack of support from the existing, formal processes. In this paper, we argue that all new concepts need to be conceptualized before being introduced to the NPD process, but that does not always suffice. Concepts other than technology concepts also need a contingent package to enable an evaluation in the context of the R&D process – they need to be contextualized. This paper draws on an in-depth case study of Volvo Cars within a long-lasting collaborative research setup. It is based on an interview study with key persons in the areas of concept work and NPD, and uses an insider/outsider approach.     

Concept learning by direct current design challenges in secondary education

This paper presents a mixed methods study in which 77 students and 3 teachers took part, that investigated the practice of Learning by Design (LBD). The study is part of a series of studies, funded by the Netherlands Organisation for Scientific Research, that aims to improve student learning, teaching skills and teacher training. LBD uses the context of design challenges to learn, among other things, science. Previous research showed that this approach to subject integration is quite successful but provides little profit regarding scientific concept learning. Perhaps, when the process of concept learning is better understood, LBD is a suitable method for integration. Through pre- and post-exams we measured, like others, a medium gain in the mastery of scientific concepts. Qualitative data revealed important focus-related issues that impede concept learning. As a result, mainly implicit learning of loose facts and incomplete concepts occurs. More transparency of the learning situation and a stronger focus on underlying concepts should make concept learning more explicit and coherent.     

The Role of Future‐Market Focus in the Early Stages of NPD across Varying Levels of Innovativeness

The concept of future‐market focus (FMF) arose out of the debate about firm size and incumbency in the face of radical or disruptive innovations, and has been demonstrated to have a positive correlation with radical innovation (RI) success. This study examines the relationship between FMF and the processes used in the early stages of NPD for four types of innovation projects: incremental innovations, technological breakthroughs, market breakthroughs, and radical innovations. We found that the future‐market focus of a project team can influence the early stage processes used in a new product innovation project, and does so differentially across levels of innovativeness. In particular, the concept generation process, understanding of market needs, and screening decision criteria are different for low‐ versus high‐FMF projects and there are differences based on the level of innovation. In addition, we found that radical innovation projects rated low in FMF are markedly different than the radical innovation projects described in prior studies.     

An Exploratory Study of the Innovation Evaluation Process

In their search for the keys to successful product innovation, product managers and researchers typically focus on trying to identify the most effective organizational processes, strategies, and structures. Surprisingly, little or no effort is directed toward understanding the process that consumers use for evaluating an innovation. By gaining insight into this evaluation process, a firm can present an innovative product in a more effective manner and thus increase the likelihood that consumers will respond favorably to the innovation. Richard W. Olshavsky and Richard A. Spreng provide insight into this process by describing the results of an experiment in which subjects were asked to evaluate several innovative concepts. From their observations, they develop a model of the detailed information-processing steps that these consumers employed in order to evaluate the new products. Consistent with previous research, they found that judgment was the predominant evaluation strategy, particularly for the most innovative concepts. Various subjects also used a categorization strategy, though none used categorization for more than four of the nine concepts that were evaluated. Contrary to expectations, none of the evaluations relied solely on the manufacturer's reputation or the recommendation of a friend. In a simplified model of the evaluation process, when presented with an innovative concept, consumers first attempt to categorize the product. In other words, an innovation may be rejected simply because consumers somehow link it to an existing category that has a negative connotation. If consumers cannot categorize the product, they then employ a judgment process based on some evaluative criteria. Based on the data collected in this study, this simplified model is extended to include four other cognitive processes that strongly influence the evaluation process: forming evaluative criteria, forming expectations about the innovative concept, assessing satisfaction with an old product, and comparing the new and old products. When faced with a highly innovative concept, consumers may find it difficult to form their own evaluative criteria and expectations concerning that innovation. Consequently, managers may have an opportunity to shape the judgment process by educating consumers about the appropriate evaluative criteria or by clearly communicating the product's attributes, benefits, and appropriate use.     

“价值主张”概念解析与未来展望

系统梳理和辨识了价值主张概念,将价值主张概念内涵归纳为“产品营销口号”、“感知承诺”、“互惠承诺”、“公司定位陈述”、“企业家远见”五类,并将五种类别的价值主张概念进行横向比较和纵向演进分析,阐明它们彼此之间的区别和联系。在此基础上,重构了价值主张概念,提出了价值主张层级模型。文末对价值主张的未来研究方向进行了展望。     

Comparing Lead Users to Emergent‐Nature Consumers as Sources of Innovation at Early Stages of New Product Development

The large potential of lead users (LUs) in developing innovative and radically new product concepts is well established in the literature. However, in a widely acknowledged study, Hoffman et al. introduced the new concept of emergent‐nature consumers (ENCs) and showed the superiority of this group of individuals over LUs in developing new product concepts. Consequently, they postulated ENCs to be “the right consumers” to be integrated in new product development processes. In this article, we critically reflect and build on Hoffman et al.'s study and further investigate the promise of the ENC concept as compared to the LU concept. In a pilot study, we replicated the study by Hoffman and colleagues: We conducted a crowdsourcing competition and asked for concepts for new services; those concepts had been generated collectively by the participants and had been assessed by a consumer crowd. In the main study, the participants of a crowdsourcing competition submitted individually generated concepts for products, which were evaluated by industry experts. Across both studies, using different empirical methods in two different contexts, and in contrast to Hoffman et al.'s work, we find support for LUs outperforming ENCs (as well as average users) in generating the commercially most promising concepts. Thus, our insights reinforce the existing user innovation literature and the notion of LUs being the primary source of new product/service concepts.     

Differential Effects of the Primary Forms of Cross Functional Integration on Product Development Cycle Time

In markets characterized by high rates of technological and market change product life cycles tend to be shorter, resulting in the increased importance of competing on the basis of product development cycle time. For firms operating in these dynamic market environments, competing on the basis of cycle time may not only be a source of competitive advantage, but in some industries may actually be essential for survival.
In this investigation the relative importance of five forms of cross functional integration and R&D integration of information or knowledge from past projects were explored in terms of their effects on product development cycle time. The five forms of cross functional integration included R&D/marketing integration, R&D/customer integration, R&D/manufacturing integration, R&D/supplier integration, and strategic partnerships. A sample of 65 U.S. and Scandinavian high technology firms (or strategic business units) were studied. The sample included firms from the computer, telecommunications, instruments, specialty chemicals, biotechnology, and software industries.
The results demonstrated that R&D integration of knowledge from past projects explained the largest degree of variation in product development cycle time. R&D/marketing integration and R&D/customer integration explained the next largest degree of variation in cycle time reduction. Cross cultural generalizability tests demonstrated that the results were generalizable across the U.S. and Scandinavian samples of firms. In addition, the results were found to be generalizable across industry or product category for five of the six forms of integration.     

Product innovation with vertical differentiation: Is a monopolist's incentive weaker?

This paper compares the incentives for product innovation across different market structures when the new product is vertically differentiated and of lower quality, a common case empirically. We show that innovation incentive rankings across market structures can differ substantially when the new product is of lower rather than higher quality. In particular, the incentive to add the new product can be greater for a monopolist over the old product than for a firm that would face any degree of competition from the old product. This incentive ranking cannot occur when, instead, the new product is of higher quality as has been analyzed in previous work. Moreover, in that case, the incentive ranking is the same whether the market is covered or not covered, whereas in our setting the ranking can differ. With the market covered, our setting provides another environment where the monopolist can have the greatest incentive to innovate, as previously shown when the new product is horizontally differentiated. Together, both settings show that Arrow's famous result—a secure monopolist gains less from a nondrastic process innovation than would a competitive firm—does not always extend to nondrastic product innovations. However, in all the cases analyzed here, consumer welfare (though not total welfare) is always lower under monopoly, even when only the monopolist would add the new product.     

How Concept Knowledge Affects Concept Evaluation

Conventional concept testing methodologies are based on the simple assumption that respondents understand the concept when they offer their evaluation of it. Obviously it's an assumption that often will not hold. What happens to the quality of the evaluation at different levels of concept knowledge is the issue addressed in this article by Eric Reidenbach and Sharon Grimes. Their research shows that the level of concept knowledge affects certain elements of the evaluation more than others, and that the evaluation is moderated by the type of innovation and the way it is presented.