Design and evaluation of ranking-type Delphi studies using best-worst-scaling

The ranking-type Delphi method has received widespread use in a large number of research areas. In order to rank a set of items, ranking-type Delphi studies use different ranking approaches which may be biased. This is intensified by the fact that the number of issues a participant can reasonably rank is limited. This paper describes an approach for the design and evaluation of ranking-type Delphi studies using best-worst-scaling (BWS). The statistical BWS design is based on a balanced incomplete block design to construct the comparison sets. The statistical evaluation encompasses the measurement of stability and consensus with the use of the coefficient of variation. Overall, our approach (1) enables to design a ranking with up to 22 items in a manner that participants can reasonably rank, (2) avoids and overcomes shortcomings of common ranking approaches, and (3) offers a statistical procedure for an unambiguous evaluation.