You Cannot Judge a Book by Its Cover: The Problems with Journal Rankings

Abstract

Journal rankings lists have impacted and are impacting accounting educators and accounting education researchers around the world. Nowhere is the impact positive. It ranges from slight constraints on academic freedom to admonition, censure, reduced research allowances, non-promotion, non-short-listing for jobs, increased teaching loads, and re-designation as a non-researcher, all because the chosen research specialism of someone who was vocationally motivated to become a teacher of accounting is, ironically, accounting education. University managers believe that these journal ranking lists show that accounting faculty publish top-quality research on accounting regulation, financial markets, business finance, auditing, international accounting, management accounting, taxation, accounting in society, and more, but not on what they do in their ‘day job’ – teaching accounting. These same managers also believe that the journal ranking lists indicate that accounting faculty do not publish top-quality research in accounting history and accounting systems. And they also believe that journal ranking lists show that accounting faculty write top-quality research in education, history, and systems, but only if they publish it in specialist journals that do not have the word ‘accounting’ in their title, or in mainstream journals that do. Tarring everyone with the same brush because of the journal in which they publish is inequitable. We would not allow it in other walks of life. It is time the discrimination ended.     

Benchmarks for evaluating the research productivity of accounting faculty

This study reports comprehensive data on both the quantity and quality of research productivity of 3878 accounting faculty who earned their accounting doctoral degrees from 1971 to 1993. Publications in 40 journals were used to measure faculty publication quantity. Journal ratings derived from a compilation of the rankings of five prior studies and co-authorship were used to measure publication quality. Choosing benchmarks for an individual faculty requires users of our data to determine four parameters: (1) what credit to give a faculty member for co-authored articles; (2) what level of journal quality is appropriate, e.g. presenting benchmarks for publications in the Best 4, Best 12, Best 22 and Best 40 journals; (3) choosing appropriate levels of performance, e.g. considering the publication record in the top 10%, top 20%, top 25%, top 33%, or top 50% of all faculty; and (4) deciding the emphasis to place on the number of years since the doctoral degree was earned. We believe that this is the first set of benchmarks that allows administrators to state, with some justification, a required number of articles for tenure or promotion. In addition, we discovered that the average number of authors per article is significantly correlated with time and growing at a pace of 0.017 authors per article per year.     

The incidence of documented standards for research in departments of accounting at US institutions

Evidence suggests that standards for research in accounting are vague to junior faculty at the same time business schools are placing more emphasis on scholarship when evaluating faculty for tenure and promotion (T&P). In response, we investigate the incidence of accounting-specific documented standards for research in T&P decisions based on an email survey of accounting department administrators at US institutions. In addition, we report respondent data about the use of documented and informal journal lists. Our findings suggest that few accounting departments, regardless of accreditation status, utilize department-specific written scholarship standards or journal lists, supporting faculty perceptions that scholarship requirements for T&P are vague. As part of our analysis we review implications of the Final Report of the AACSB International Impact of Research Task Force (AACSB International, 2008) on the use of journal lists for tenure and promotion decisions. We summarize by advocating for specific accounting department-level policies for T&P, including consideration of explicit journal lists.     

Journal Rankings,Collaborative Research and Publication Strategies: Evidence from China

Abstract

The primary focus of our paper is on the potential for in-house journal ranking lists to create friction between international collaborating researchers due to differences in how particular journals are rated on different lists. Using a questionnaire distributed to Chinese accounting researchers, we identify a number of potential friction points between Chinese and UK researchers. We find that almost all of our Chinese respondents use their own school's in-house ranking list as the primary or exclusive reference point for assessing journal quality, and 73% of respondents acknowledge that this has caused problems when working with scholars from other universities because of differences in how their institutions rank journals.     

Ranking accounting journals using dissertation citation analysis: A research note

Prior literature on accounting journal rankings has provided different journal lists depending on the type of examination (citations- vs. survey-based) and the choice of journals covered. A recent study by Bonner, Hesford, Van der Stede, and Young (2006) [Bonner, S., Hesford, A., Van der Stede, W. A., & Young, M. S. (2006). The most influential journals in academic accounting. Accounting, Organizations and Society, 31(7), 663–685] documents disproportionately more citations in the financial accounting area, suggesting a financial accounting bias in the accounting literature. We use citations from accounting dissertations completed during 1999–2003 to provide a ranking of accounting journals. The database allows us to assess the research interests of new accounting scholars and the literature sources they draw from. Another innovation is our ranking of accounting journals based on specialty areas (auditing, financial, managerial, tax, systems, and other) and research methods (archival, experimental, modeling, survey, and other). To mitigate the financial accounting bias documented by Bonner et al. (2006), we derive a ranking metric by scaling (normalizing) the journal citations by the number of dissertations within each specialty area and research method. Overall, the top journals are, JAR, AOS, TAR, and JAE. We also provide evidence that top journal rankings do vary by specialty area as well as by research methods.     

Accounting Journal Rankings,Authorship Patterns and the Author Affiliation Index

We conduct an evaluation of 43 accounting journals using the author affiliation index (AAI). Our results suggest that the Australian Business Dean's Council (ABDC) ratings are consistent with the AAI‐based rankings. Nonetheless, there are a few highly (lowly) regarded accounting journals in terms of AAI receiving a relatively lower (higher) rating in the ABDC journal ranking list. The co‐authorship patterns suggest that top AAI and near‐top AAI journals actually see more co‐authorship from scholars in top programs and scholars in other programs (both ranked 21–100 and ‘others’).     

Accounting journals and the ABS quality ratings

This study aims to look behind the quality ratings for accounting journals, listed in the ABS Academic Journal Quality Guide (Kelly, Morris, Rowlinson and Harvey, 2009). Significant variations exist in the perceptions of journal quality across the six UK business schools that contribute ratings to the ABS guide, with the most optimistic perceptions tending to come from those schools whose ratings are more highly correlated with quality scores for critical and interpretive research, as reported in Lowe and Locke's (2005) article in Accounting, Organizations and Society (30:1, 81–98). Pessimistic perceptions are more likely to exist in those schools whose ratings are more highly correlated with Lowe & Locke's scores for functional and capital markets research paradigms. There are also notable variations in journal ratings across time. Given that perceptions vary so much across schools, paradigms and time, how much credence can be given to any single rating system for journal quality? This study concludes that if the ABS guide is to be used by university decision-makers or heads of school then the ABS ratings for any given year need to be treated with extreme caution and with an appropriate recognition of their intrinsic limitations.     

Social accounting research: An Australasian perspective

This paper provides an insight into the nature and the extent of social accounting research being undertaken within Australasia. It demonstrates that Australasian researchers account for a significant amount of internationally published social accounting research, but emphasises that the research effort seems to be confined to a limited number of researchers perhaps reflecting a lack of ‘take-up’ in this area in terms of the scale of participation. Information is also presented about the relative propensity of journals within the sample to publish social accounting research, and identifies that ‘top tier’ accounting journals historically have not published social accounting research. The paper also considers various factors which seem to be impeding the ‘recruitment’ of new social accounting researchers.     

The most influential journals in academic accounting

In this article we summarize the findings of articles that have ranked academic accounting journals, as well as articles that provide other bases for considering journal quality. Results indicate that five journals—Accounting, Organizations and Society, Contemporary Accounting Research, Journal of Accounting and Economics, Journal of Accounting Research, and The Accounting Review—rank consistently as the top journals in the field. However, these five journals differ substantially as to the numbers of articles they publish overall as well as the proportions of articles that are related to the various specialty areas of accounting. Further, the relative proportions of articles by area do not correspond to the numbers of individuals working in the specialty areas. Financial accounting articles appear in disproportionately high numbers for all journals except Accounting, Organizations and Society, whereas management accounting articles appear in disproportionately low numbers for all journals except Accounting, Organizations and Society. In all journals, systems and tax articles also appear to be disproportionately low vis-à-vis the numbers of individuals working in these areas. Auditing receives fairly even exposure across journals and vis-à-vis individuals in the area, except in the Journal of Accounting and Economics.     

Ranking Journals Using Social Science Research Network Downloads

I use a new approach to rank journals, namely the number and percent frequency of articles a journal publishes that are heavily downloaded from the Social Science Research Network (SSRN). I rank 18 accounting and finance journals, and I identify five journals not considered by the two most recent major published ranking studies of publications by accounting faculty, namely (in rank order): Journal of Financial Economics, Review of Accounting Studies, Review of Quantitative Finance and Accounting, Journal of Corporate Finance, and Journal of International Financial Management and Accounting. I show that financial accounting faculties are more likely to post their working papers to SSRN, and papers posted by financial faculties generate more downloads. I mitigate this bias in favor of the financial area by providing separate rankings based on authors in the financial versus non-financial areas.     

Faculty perceptions of journals: quality and publishing feasibility

Tenure-track faculty at AACSB-accredited colleges were surveyed regarding their perceptions of 152 journals. Response measures included perceptions of journal quality and the feasibility of publishing in each journal. Analyses of responses from 616 faculty document statistically significant differences in both quality and publishing feasibility across journals, scholarship areas, and degree-granting categories (doctoral versus nondoctoral). Significant interactions were also found to exist across these factors. Effect size estimates for variations in quality and feasibility across journals, scholarship areas, and the interaction of journals and scholarship areas suggest that the magnitudes of observed differences are nontrivial. Listings of the 20 highest quality journals for most individual scholarship areas were found to have little in common with an overall top-20 listing. Overall, these results suggest that area-specific journal ratings provide better information than a single overall ranking list.     

Food for Thought on the ABS Academic Journal Quality Guide

This paper discusses issues relating to the use of the Association of Business Schools' (ABS) Academic Journal Quality Guide within UK business schools. It also looks at several specific issues raised by the Chair of the British Accounting Association/British Accounting and Finance Association regarding the ratings for top international journals, and for accounting education and accounting history journals. The increasing use of this guide by business school deans/heads as a tool for staffing and research resource allocation has significant implications both for individuals and specialist areas of research.     

Using Google Scholar citations to rank accounting programs: a global perspective

We conduct an assessment on accounting program research performance based on Google Scholar citations for all articles from a set of 23 quality accounting journals during 1991–2010. Our work is a new approach in accounting by directly measuring the impact of the faculty research in accounting programs. We find that the top-5 accounting programs are the University of Pennsylvania, the University of Chicago, Stanford University, the University of Michigan, and Harvard University. These top programs produce a large number of high impact articles. In addition, using the mean citations from all articles in a journal, we find that the Review of Accounting Studies (RAST) is a top-5 journal, replacing Contemporary Accounting Research (CAR).     

BIAS IN JOURNAL RATINGS

The present study investigated the association between faculty publication records and their point-based evaluations of finance journals. No relationship was detected between the merit points assigned to finance journals and the journal-specific success of the faculty rendering the journal ratings. However, a negative relationship was found between general publication success of faculty and the merit points they assigned to lower-level journal publications. The association was particularly strong for faculty who had published in the top three finance journals.     

Setting realistic expectations for publishing in leading accounting research journals

At most colleges and universities, an accounting academic's publication record is a primary consideration in promotion and tenure decisions. Many institutions encourage and expect faculty members to publish in leading accounting research journals. No longer limited to institutions with strong research orientations, these expectations often become an informal part of promotion and tenure guidelines at other colleges and universities. This article evaluates the use of such criteria in light of a five-year study (1978–1982) of several attributes of academics who have published in two leading research journals, The Accounting Review and The Journal of Accounting Research. The objective is to initiate dialogue and promote reasonable expectations among and between accounting faculty and administrators.     

Producing Spaces for Academic Discourse: The Impact of Research Assessment Exercises and Journal Quality Rankings

This study examines the impact of national research assessment exercises (NRAEs) and associated journal quality rankings on the development, scope and sustainability of the academic journals in which accounting research is disseminated. The reported exploratory study focused on the United Kingdom (UK), Australia and New Zealand as three countries in which NRAEs are well developed or imminent. Data were collected via a survey of authors, interviews with journal editors, and feedback from publishers responsible for producing academic accounting journals.
The findings suggest that, despite cynicism around the reliability of published journal quality rankings, the entrenchment of NRAE 'rules' and journal quality perceptions has changed authors' submission choices and left lower ranked journals struggling with a diminished quantity and quality of submissions. A clear perception is that NRAEs have done little to improve the overall quality of the accounting literature, but are impeding the diversity, originality and practical relevance of accounting research.
Although strategies are suggested for meeting these challenges, they require strategic partnerships with publishers to enhance the profile and distribution of emerging journals, and depend on the willingness of accounting researchers to form supportive communities around journals that facilitate their research interests. The alternative may be a withering of the spaces for academic discourse, a stifling of innovation and a further entrenchment of current perceptions of what counts as 'quality' research.     

Productivity in “Top‐Ten” Academic Accounting Journals by Researchers at Canadian Universities*

We examine the research productivity of academic accountants at Canadian universities for the 11‐year period 1990‐2000. Our analysis is based on the “top‐ten” ranked refereed journals in accounting, auditing, and taxation, as documented by Brown and Huefner (1994). We first provide an overview of the importance of publishing in highly ranked accounting journals for individual academics, departments, and business faculties. We then provide details of the proportion of articles published in each of these journals by academics from Canadian universities; the type of research published in each journal (auditing, financial accounting, managerial accounting, and taxation); and details of editorial board service. Our results indicate that even at the most productive Canadian university (in terms of “top‐ten” publications), faculty members publish (on average) approximately one article every seven years. Six Canadian universities have faculty members with, on average, more than one article in “top‐ten” journals every 10 years. We also provide results of analyses that rank each Canadian university, after controlling for the relative quality of each journal, using impact factors published by the Social Science Citation Index. In addition, statistics are provided with regard to the 15 most productive researchers, in terms of “top‐ten” publications, in the 11‐year period. Finally, in conjunction with the 25th anniversary of the Canadian Academic Accounting Association, we examine the productivity of academic accountants at Canadian universities over the past 25 years by combining our results with those reported by Richardson and Williams (1990).     

On the Optimal Relation between the Properties of Managerial and Financial Reporting Systems

We develop a theoretical model of the firm that links properties (stewardship vs. valuation focus) of financial reporting regimes with the informational properties of optimal managerial accounting systems. We show that, contrary to the standard textbook proposition, properties of management and financial accounting systems are not independent. Significantly, we provide an explicit connection between exogenous and observable properties of a firm's financial reporting system and the quality of the managerial accounting system on which manager(s) base real economic decisions. As the quality of those economic decisions can also be inferred from publicly available data, our theory generates new opportunities for empirical managerial accounting research on large nonproprietary samples. Further, by being able to identify enhanced performance due to improved managerial accounting information, our theory provides opportunities to gain a better understanding of the link between particular managerial accounting practices and the quality of the information produced.     

Reproducibility in Accounting Research: Views of the Research Community

We have little knowledge about the prevalence of irreproducibility in the accounting literature. To narrow this gap, we conducted a survey among the participants of the 2019 JAR Conference on their perceptions of the frequency, causes, and consequences of irreproducible research published in accounting journals. A majority of respondents believe that irreproducibility is common in the literature, constitutes a major problem, and receives too little attention. Most have encountered irreproducibility in the work of others (although not in their own work) but chose not to pursue their failed reproduction attempts to publication. Respondents believe irreproducibility results chiefly from career or publication incentives as well as from selective reporting of results. They also believe that practices like sharing code and data combined with stronger incentives to replicate the work of others would enhance reproducibility. The views of accounting researchers are remarkably similar to those expressed in a survey by the scientific journal Nature. We conclude by discussing the implications of our findings and provide several potential paths forward for the accounting research community.