Evaluating rater training with double-pretest one-posttest designs: an analysis of testing effects and the moderating role of rater self-efficacy

Abstract

Previous research on performance appraisal in Human Resource Management has found that frame-of-reference (FOR) training can improve rating accuracy. However, both the time-consuming development of training materials and the almost exclusive use of experimental evaluation designs limit the dissemination of FOR training as well as a better understanding of how and when it works. Therefore, unlike past research, the present studies used more general rater training materials and examined improvements in rater accuracy by means of a double-pretest one-posttest design. Study 1, using a student sample (the majority being part-time employees; N = 58), demonstrated the effectiveness of the rater training over and above a testing effect. Study 2 with participants from the workforce population (N = 45) replicated these results. In addition, in Study 2, pre-training rater self-efficacy moderated rater training effectiveness such that the rating accuracy of trainees with low self-efficacy improved more than did accuracy scores of trainees high in rater self-efficacy. We conclude that an effective FOR training can be conducted with less organization-specific materials, though both testing effects must be controlled for and individual differences (i.e. rater self-efficacy) between trainees have to be taken into account.