You Cannot Judge a Book by Its Cover: The Problems with Journal Rankings

Abstract

Journal rankings lists have impacted and are impacting accounting educators and accounting education researchers around the world. Nowhere is the impact positive. It ranges from slight constraints on academic freedom to admonition, censure, reduced research allowances, non-promotion, non-short-listing for jobs, increased teaching loads, and re-designation as a non-researcher, all because the chosen research specialism of someone who was vocationally motivated to become a teacher of accounting is, ironically, accounting education. University managers believe that these journal ranking lists show that accounting faculty publish top-quality research on accounting regulation, financial markets, business finance, auditing, international accounting, management accounting, taxation, accounting in society, and more, but not on what they do in their ‘day job’ – teaching accounting. These same managers also believe that the journal ranking lists indicate that accounting faculty do not publish top-quality research in accounting history and accounting systems. And they also believe that journal ranking lists show that accounting faculty write top-quality research in education, history, and systems, but only if they publish it in specialist journals that do not have the word ‘accounting’ in their title, or in mainstream journals that do. Tarring everyone with the same brush because of the journal in which they publish is inequitable. We would not allow it in other walks of life. It is time the discrimination ended.     

Ranking accounting journals using dissertation citation analysis: A research note

Prior literature on accounting journal rankings has provided different journal lists depending on the type of examination (citations- vs. survey-based) and the choice of journals covered. A recent study by Bonner, Hesford, Van der Stede, and Young (2006) [Bonner, S., Hesford, A., Van der Stede, W. A., & Young, M. S. (2006). The most influential journals in academic accounting. Accounting, Organizations and Society, 31(7), 663–685] documents disproportionately more citations in the financial accounting area, suggesting a financial accounting bias in the accounting literature. We use citations from accounting dissertations completed during 1999–2003 to provide a ranking of accounting journals. The database allows us to assess the research interests of new accounting scholars and the literature sources they draw from. Another innovation is our ranking of accounting journals based on specialty areas (auditing, financial, managerial, tax, systems, and other) and research methods (archival, experimental, modeling, survey, and other). To mitigate the financial accounting bias documented by Bonner et al. (2006), we derive a ranking metric by scaling (normalizing) the journal citations by the number of dissertations within each specialty area and research method. Overall, the top journals are, JAR, AOS, TAR, and JAE. We also provide evidence that top journal rankings do vary by specialty area as well as by research methods.     

Publication outlets for healthcare financial research: A study of journal quality perceptions

Publications in high quality journals often serve to indicate research productivity. However, many top‐rated journals infrequently publish cross‐disciplinary topics such as healthcare financial management (HFM). So, academic administrators and HFM researchers find it challenging to evaluate the quality of the work. Journals open to publishing HFM articles are distributed across multiple disciplines. Each field has its own journal rankings typically focused on their primary subject area. Starting with prior literature, we form a cross‐disciplinary journal list and solicit preliminary input from editors, associated editors, reviewers, and authors. We then solicit confirmatory ranking input from independent researchers in a holdout sample.     

Author Affiliation Index, finance journal ranking, and the pattern of authorship

In this paper we use a new method to rank finance journals and study the pattern of authorship/co-authorship across journals. Defined as the ratio of articles authored by faculty at the world's top 80 finance programs to the total number of articles by all authors, the Author Affiliation Index is a cost-effective and intuitively easy-to-understand approach to journal rankings. Forty-one finance journals are ranked according to this index. If properly constructed, the Author Affiliation Index provides an easy and credible way to supplement the existing journal ranking methods. Our ranking system reveals the journal–researcher clientele, and we find that collaboration (co-authoring) between faculty within elite programs exists only in top-tier and near-top-tier journals. Publications in lower-tier journals by researchers of elite programs are driven by their co-authors. Collaboration between faculty in elite and non-elite programs, however, is more prevalent than that within elite programs across all tiers of journals. Co-authorship among top 80 programs, nevertheless, is more common in top-tier journals, while co-authorship between top 80 and other programs is more dominant in lower-ranked journals.     

The demise of the ARC journal ranking scheme: an ex post analysis of the accounting and finance journals

The recent abolition of the ARC journal ranking scheme is indicative of some problematical features of journal ranking in general and the ARC scheme in particular. An alternative citation‐based ranking scheme is applied to the accounting and finance journals to highlight some loopholes in the abandoned ARC scheme and provide some suggestions for how to proceed with ERA 2012. By re‐ranking journals according to their citation indices, it is demonstrated that the ARC ranking placed a large number of journals where they do not belong. As a result, the ARC scheme induced some adverse behavioural changes with respect to preferred publication outlets.     

Ranking accounting authors and departments in accounting education: different methodologies – significantly different results

This research tests whether Holderness Jr., D. K., Myers, N., Summers, S. L., & Wood, D. A. [(2014). Accounting education research: Ranking institutions and individual scholars. Issues in Accounting Education, 29(1), 87–115] accounting-education rankings are sensitive to a change in the set of journals used. It provides updated rankings for accounting-education authors from Australia, Canada, New Zealand, the Republic of Ireland, the United Kingdom, and the United States using a sample that included the publications in 13 accounting-education journals. Our analysis indicated that Holderness et al.’s rankings of authors and departments were significantly different from our rankings. This research provides rankings of the top 50 authors and departments for three periods: from 2010 to 2015, from 2004 to 2015, and from 1992 to 2015. We provide data indicating the distribution of authors for these periods to assist authors not listed in the most prolific lists in determining their relative ranking. Finally, we provide data on the distribution of journal choices for accounting-education publications for the authors from each country.     

Reproducibility in Accounting Research: Views of the Research Community

We have little knowledge about the prevalence of irreproducibility in the accounting literature. To narrow this gap, we conducted a survey among the participants of the 2019 JAR Conference on their perceptions of the frequency, causes, and consequences of irreproducible research published in accounting journals. A majority of respondents believe that irreproducibility is common in the literature, constitutes a major problem, and receives too little attention. Most have encountered irreproducibility in the work of others (although not in their own work) but chose not to pursue their failed reproduction attempts to publication. Respondents believe irreproducibility results chiefly from career or publication incentives as well as from selective reporting of results. They also believe that practices like sharing code and data combined with stronger incentives to replicate the work of others would enhance reproducibility. The views of accounting researchers are remarkably similar to those expressed in a survey by the scientific journal Nature. We conclude by discussing the implications of our findings and provide several potential paths forward for the accounting research community.     

Exploring the Role of Symbolic Legitimation in Voluntary Journal List Adoption

Abstract

Journal lists are common in the global accounting academy, yet the underlying ‘research quality’ construct is ill defined and the specific processes used in voluntary list adoptions have not been systematically tied to theory. This paper raises the questions: (1) What is driving voluntary journal list adoptions? and (2) Are journal list projects more closely aligned with addressing well-specified public problems or with effecting symbolic legitimation? A set of theory-based questions help structure an assessment of the process used by two American accounting/business programs with traditional teaching missions as they work toward developing an institution-specific journal list. Subject to the usual cautions about the generalizability of a case study described by an active player, the narrative suggests that symbolic legitimation issues may play a key role in journal listing and ranking processes.     

Faculty perceptions of journals: quality and publishing feasibility

Tenure-track faculty at AACSB-accredited colleges were surveyed regarding their perceptions of 152 journals. Response measures included perceptions of journal quality and the feasibility of publishing in each journal. Analyses of responses from 616 faculty document statistically significant differences in both quality and publishing feasibility across journals, scholarship areas, and degree-granting categories (doctoral versus nondoctoral). Significant interactions were also found to exist across these factors. Effect size estimates for variations in quality and feasibility across journals, scholarship areas, and the interaction of journals and scholarship areas suggest that the magnitudes of observed differences are nontrivial. Listings of the 20 highest quality journals for most individual scholarship areas were found to have little in common with an overall top-20 listing. Overall, these results suggest that area-specific journal ratings provide better information than a single overall ranking list.     

Finance journal rankings and tiers: An Active Scholar Assessment methodology

This study uses respondent data from a web-based survey of active finance scholars (45% response rate from 37 countries) to endogenously rank 83 finance journals by quality and importance. Journals are further tiered into four groups (A, B, C and D) and stratified into “upper”, “middle” and “lower” tier categories (e.g. A+, A and A−) by estimating a nested regression with random journal-within-tier effects. The comprehensive and endogenous ranking of finance journals based on the Active Scholar Assessment (ASA) methodology can help authors evaluate the strategic aspects of placing their research, facilitate assessment of research achievement by tenure and promotion committees; and assist university libraries in better managing their journal resources. Study findings from active researchers in the field also provide useful guidance to editorial boards for enhancing their journal standing.     

Aggregating Incomplete Lists of Journal Rankings: An Application to Academic Accounting Journals*

We introduce a branch‐and‐cut algorithm to aggregate published journal rankings based on subsets of the accounting literature in order to create a consensus ranking. The aggregate ranking allows specialist and regional journals, which may only be ranked in a limited number of studies, to be placed with respect to each other and with respect to the generalist journals that are usually included in ranking studies. The approach we develop is a significant advance over ad hoc approaches to aggregating journal rankings that have appeared in the literature and may provide a theoretically sound and replicable basis for further exploration of the concept of journal quality and the stability of journal rankings over time and ranking methods.     

Ranking Journals Using Social Science Research Network Downloads

I use a new approach to rank journals, namely the number and percent frequency of articles a journal publishes that are heavily downloaded from the Social Science Research Network (SSRN). I rank 18 accounting and finance journals, and I identify five journals not considered by the two most recent major published ranking studies of publications by accounting faculty, namely (in rank order): Journal of Financial Economics, Review of Accounting Studies, Review of Quantitative Finance and Accounting, Journal of Corporate Finance, and Journal of International Financial Management and Accounting. I show that financial accounting faculties are more likely to post their working papers to SSRN, and papers posted by financial faculties generate more downloads. I mitigate this bias in favor of the financial area by providing separate rankings based on authors in the financial versus non-financial areas.     

Measuring the impact of accounting journals using Google Scholar and the g-index

The UK's proposed Research Excellence Framework promotes a move towards citation analysis for assessing research performance. However, for business disciplines, journal rankings are likely to remain an important aid in evaluating research quality. The accounting literature includes many journal rankings and citation studies, however there has been little coverage of recent advances in these areas. This study explores approaches to assessing the impact of accounting journals with a focus on quantitative measures as a complement to peer-review-based evaluation. New data sources and techniques for citation studies are reviewed, and the g-index is selected for further analysis. The g-index was developed by Professor Leo Egghe in 2006 as an improvement on the h-index. Like the h-index, the g-index represents a relationship between papers published and the level of citations they receive, but the g-index is more sensitive to highly cited paper. To apply the g-index to accounting journals, the study first combines eight published journals rankings to produce a list of 34 highly-regarded titles. Citation data are then gathered from Google Scholar and used to calculate g-index scores as the basis of a new ranking. Google Scholar is found to have broader coverage of accounting citations than Scopus or the Web of Science databases, but requires cleaning to remove duplicate entries. The use of the g-index for ranking journals is found to be a useful innovation in citation analysis, allowing a more robust assessment of the impact of journals.     

A half-century of Accounting and Business Research: the impact on the study of international financial reporting

This paper celebrates the contribution of this journal, over its first 50 years, to research on international financial reporting, defined as comprising writings on comparative or harmonisation topics. The paper examines the journal’s output in that field and how it contributed to the field’s development. Even though the journal was sympathetic to international financial reporting, less than 1% of output in its first decade (the 1970s) related to it. In its first 35 years, a large proportion of the journal’s limited output in the field was produced by two small groups of researchers. However, during its fourth decade, the field gradually became dominant as the accounting world changed. By then, the journal had already published the seminal papers on several central topics in international financial reporting, including measuring harmonisation, using reconciliations to measure international accounting differences, assessing international differences in the influence of tax on financial reporting, and measuring international difference in the application of international standards. These topics were later taken up by many researchers in several other journals.     

An Analysis of Risk, Insurance, and Actuarial Research: Citations From 1996 to 2000

This article evaluates the relative significance of research published in 16 risk, insurance, and actuarial journals by examining the frequency of citations in these risk, insurance, and actuarial journals and 16 of the leading finance journals during the years 1996 through 2000. First, the article provides the frequency with which each sample risk, insurance, and actuarial journal cites itself and the other sample journals so as to communicate the degree to which each journal's published research has had an influence on the other sample journals. Then the article divides the 16 journals into two groups: (1) the risk and insurance journal group, and (2) the actuarial journal group, and ranks them within their group based on their total number of citations, including and excluding self‐citations. A ranking within each group is based on the journals’ influence on a per article published basis. Finally, this study observes and reports on the most frequently cited articles from the sample risk, insurance, and actuarial journals.     

Ranking North American accounting scholars publishing accounting education papers: 1966–2011

This paper ranks accounting’s education authors who teach at institutions located in the United States and Canada. During the 46-year period from 1966 through 2011 that we examined, 13 journals published accounting education papers; the publication period for each journal varies. The data indicate that only 31.4% of accounting’s 4855 doctoral faculty who teach at schools in North America have one or more publications in these 13 journals. For those doctorates still teaching, the research provides rankings of authors by doctoral year and for four periods: 2002–2011 (most recent 10 years), 1992–2001 (next 10-year period), 1966–1991 (last 26 years), and for the entire 46-year period. To acknowledge the contributions of retired and deceased authors, the research lists those authors who would have been included on the overall list had they still been actively teaching. While Urbancic (2009) and Brigham Young University (BYU) provide rankings of authors in accounting education, these rankings are limited in the scope of the journals included – Urbancic includes only six accounting education journals, while BYU includes only Issues in Accounting Education. We found that Urbancic’s (BYU’s) 10-year (20-year) data had a Spearman’s rho of −0.84 (0.39) with our rankings. We believe that data presented herein provides a more comprehensive ranking of accounting’s authors in the area of education.     

The incidence of documented standards for research in departments of accounting at US institutions

Evidence suggests that standards for research in accounting are vague to junior faculty at the same time business schools are placing more emphasis on scholarship when evaluating faculty for tenure and promotion (T&P). In response, we investigate the incidence of accounting-specific documented standards for research in T&P decisions based on an email survey of accounting department administrators at US institutions. In addition, we report respondent data about the use of documented and informal journal lists. Our findings suggest that few accounting departments, regardless of accreditation status, utilize department-specific written scholarship standards or journal lists, supporting faculty perceptions that scholarship requirements for T&P are vague. As part of our analysis we review implications of the Final Report of the AACSB International Impact of Research Task Force (AACSB International, 2008) on the use of journal lists for tenure and promotion decisions. We summarize by advocating for specific accounting department-level policies for T&P, including consideration of explicit journal lists.     

Accounting Journal Rankings,Authorship Patterns and the Author Affiliation Index

We conduct an evaluation of 43 accounting journals using the author affiliation index (AAI). Our results suggest that the Australian Business Dean's Council (ABDC) ratings are consistent with the AAI‐based rankings. Nonetheless, there are a few highly (lowly) regarded accounting journals in terms of AAI receiving a relatively lower (higher) rating in the ABDC journal ranking list. The co‐authorship patterns suggest that top AAI and near‐top AAI journals actually see more co‐authorship from scholars in top programs and scholars in other programs (both ranked 21–100 and ‘others’).     

A framework for journal assessment: The case of the Journal of Banking & Finance

We propose a citation-based framework for the assessment of journal influence and demonstrate its application with the Journal of Banking and Finance. The framework assesses the current position of the journal, its trends and the reasons behind the trends. The Journal of Banking and Finance is found to be one of the leading journals in the field of finance and its influence is increasing. In fact, it appears to be part of a group of leading journals that is separating itself from the other journals that are generally regarded as peers. Finally, the Journal of Banking and Finance is found to be the only leading finance journal with a high concentration of banking articles, making it a primary outlet for influential articles in this area.     

A pragmatic model to estimate journal quality in accounting

This article presents a model to estimate the relative quality of publication outlets based on objective journal characteristics. Our model improves upon the one proposed by Bean and Bernardi [Bean, D. F., & Bernardi, R. A. (2005). Estimating the ratings of journals omitted in prior quality ratings. Advances in Accounting Education, 7, 109–127.] in three important ways. First, we develop a dependent variable that is a composite score based on five prior journal perception studies. Second, our model considers different independent variables; audience, journal availability, inclusion in the Social Sciences Citation Index (an independent measure of quality), and the journal’s submission fee. This combination of variables increases the model’s explanatory power by 21% compared to Bean and Bernardi’s average R². Finally, the results of our model are more consistent with those of prior perception studies. We also apply the model to recent accounting faculty publications, which provides a comparative rating of more than 200 journals. We expect our model for estimating journal quality to help faculty, promotion and tenure committees, and university administrators evaluate the quality of journals where accounting faculty publish, an important aspect of assessing research productivity.