Evaluating Journal Quality: Beyond “Expert” Journal Assessments in the IS Discipline

Assessing the relative stature of journals devoted to the information systems (IS) discipline is an important issue for IS scholars and those who evaluate them. Even though journal assessment results are often dubiously applied by those making hiring, promotion, and merit decisions, the fact that they are so often a major ingredient in these decisions demands that we understand underlying journal assessment processes. Beyond processes involving the opinions of various “experts,” we here examine how IS journals can be evaluated based on overt behaviors of crowds of IS scholars. These behaviors are revealed preferences, in contrast to stated preferences found in opinions. Two classes of objective journal assessments are studied: impact measures and power measures. Among the former, we find that so-called journal impact factors are problematic, rendering their meaningfulness in evaluating journal stature highly suspect. Another kind of impact measure, the H-index, is found to be a more straightforward way to gauge journal impact. Two power measures for assessing IS journal stature are examined: publishing intensity and publishing breadth. The stature of IS journals according to each of the impact measures and power measures is determined. A comparison of the results shows that a small group of four or five IS journals are repeatedly found at the top across multiple objective assessment approaches. To account for both the consumption and production of IS research, it is suggested that a combined use of impact and power measures be employed in exercises aimed at evaluating relative statures of journals devoted to IS research.