Ranking journal quality by harmonic mean of ranks: an application to ISI statistics & probability

As the preponderance of journal rankings becomes increasingly more frequent and prominent in academic decision making, such rankings in broad discipline categories is taking on an increasingly important role. The paper focuses on the robustness of rankings of academic journal quality and research impact using the widely‐used Thomson Reuters ISI Web of Science citations database (ISI) for the Statistics & Probability category. The paper analyses 110 ISI international journals in Statistics & Probability using quantifiable research assessment measures (RAMs), and highlights the similarities and differences in various RAMs, which are based on alternative transformations of citations and influence. Alternative RAMs may be calculated annually or updated daily to determine when, where and how (frequently) published papers are cited (see Chang, McAleer and Oxley (2011a, b, c) , Chang, Maasoumi and McAleer (2012) ). The RAMs are grouped in four distinct classes that include impact factor, mean citations and non‐citations, journal policy, number of high quality papers, and journal influence and article influence. These classes include the most widely used RAMs, namely the classic 2‐year impact factor including journal self citations (2YIF), 2‐year impact factor excluding journal self citations (2YIF^*), 5‐year impact factor including journal self citations (5YIF), Eigenfactor (or Journal Influence), Article Influence, h‐index, PI‐BETA (Papers Ignored – By Even The Authors), 5YD2 (= 5YIF/2YIF) as a measure of citations longevity, and escalating self citations as a measure of increasing journal self citations. The paper highlights robust rankings based on the harmonic mean of the ranks of RAMs across the 4 classes. It is shown that focusing solely on the 2YIF of a journal, which partly answers the question as to when published papers are cited, to the exclusion of other informative RAMs, which answer where and how (frequently) published papers are cited, can lead to a distorted evaluation of journal quality, impact and influence relative to the more robust harmonic mean of the ranks.     

General‐interest versus specialty journals: Using intellectual influence of econometrics research to rank economics journals and articles

This paper demonstrates the potential problem in using existing economics journal rankings to evaluate the research productivity of scholars by constructing a new ranking of economics journals and articles. Based on 2142 econometrics sample articles published from 2000 to 2005, our ranking results show that the intellectual influence of an econometrics article published in several econometrics/statistics journals is much higher than if it were published in the most prestigious general‐interest journal. Given that a study's potential influence is integrated into the submission decision, this suggests a substantial downward bias toward econometricians when existing rankings are used to evaluate their research productivity. Copyright © 2009 John Wiley & Sons, Ltd.     

A Citation-Based Ranking of Strategic Management Journals

Rankings of strategy journals are important for authors, readers, and promotion and tenure committees. We present several rankings, based either on the number of articles that cited the journal or the per article impact. Our analyses cover various periods between 1991 and 2006, for most of which the Strategic Management Journal was in first place and Journal of Economics & Management Strategy (JEMS) second, although JEMS ranked first in certain instances. Long Range Planning and Technology Analysis & Strategic Management also achieve a top position. Strategic Organization makes an impressive entry and achieves a top position in 2003–2006.     

Research Quality Rankings of Heterodox Economic Journals in a Contested Discipline

This article argues that the discipline of economics consists of two subdisciplines: heterodox and mainstream economics. Being distinct bodies of knowledge, it is possible that the processes of building scientific knowledge are different enough so to generate distinctly different referencing and citation practices. Therefore, a specific impact contribution score is necessary for ranking heterodox journals in terms of their contribution to building heterodox economics. If properly developed such a metric could also be used to produce a single overall quality‐equality ranking of mainstream and heterodox journals. Utilizing citation data and peer evaluations of 62 heterodox economics journals, a research quality measure is developed and then used to rank the journals. The measure is then used in conjunction with the SSCI five‐year impact factor to produce a comparative research quality‐equality rankings of the 62 heterodox and the 192 mainstream journals in the SSCI.     

Ranking Economics Departments in a Contested Discipline: A Bibliometric Approach to Quality Equality Between Theoretically Distinct Subdisciplines

Quality ranking of economic journals and departments is a widespread practice in the United States. The methods used are peer review and bibliometric measures. In a divided discipline such as economics scientific knowledge is contested. So knowing which journals and departments are the best in terms of research is somewhat muddied. If the methods used to measure the production of quality scientific knowledge are tilted towards one of the contested approaches, the resulting quality rankings of journals and departments are tilted as well. So if the objective is the open‐minded pursuit of the production of scientific knowledge, then it is important to have measures of quality that treat the different contested approaches equally. Our article explores this issue by examining the impact that a quality‐equality bibliometric measure can have on the quality rankings of doctoral economic programs in the United States.     

Rankings Methodology for International Comparisons of Institutions and Individuals: an Application to Economics in Australia and New Zealand

Abstract.  The objectives of this paper are threefold. First, we critically survey and analyse the different methodologies that have been adopted in the rankings literature. Second, using journal articles included in the ECONLIT database and on the basis of two criteria – one based on citations and the other on perceptions of journal quality, we rank economics teaching departments in Australia and New Zealand for 1988–2002 and 1996–2002 and for individual academic economists for the periods 1988–2002, 1988–1995 and 1996–2002. Furthermore, we identify individual star performers and recognize them in a designated 'Hall of Fame' for 1988–2002. Third, our methodology enables us to make international comparisons on total and per capita bases. Previous multi-country ranking studies in economics do not rank economics departments. They provide rankings based on total publications in economics in universities regardless of whether the economists are in the economics departments or in other departments. Thus, no rankings on per capita bases are provided. We correct this shortcoming by providing total and per capita rankings and analysing the correlations between total and per capita rankings. This is the first study to examine whether the size of the economics department matters with regard to productivity.     

Rankings von Zeitschriften und Personen in der BWL

Rankings are of increasing importance in academia, but they are also criticised. In this article only research rankings are analysed, sorting either academic journals or academic authors publishing therein. Business administration and management is used as an example, whereas the analysis could be used for most social and other sciences. In business administration VHB-JOURQUAL 2 is a current example of a journal ranking, whereas the Handelsblatt ranking of management researchers is a recent ranking of persons. As an alternative to the latter one a citation-based ranking of researchers is developed and presented that possesses some advantages, especially in regard of evaluating individual academic performance, but exhibits some specific problems, too.     

WHAT MAKES A GREAT JOURNAL GREAT IN ECONOMICS? THE SINGER NOT THE SONG

Abstract This paper analyses what makes a great journal great in economics. Alternative research assessment measures (RAM) are discussed, with an emphasis on the Thomson Reuters Institute for Scientific Information (ISI) Web of Science database. ISI RAM that are calculated annually or updated daily are defined, including the classic 2‐year impact factor (2YIF), 5‐year impact factor (5YIF), immediacy (zero‐year impact factor (0YIF)), eigenfactor score, article influence, citation performance per paper online, h‐index, Zinfluence, PI‐BETA (papers ignored ‐ by even the authors) and two new RAM measures, self‐citation threshold approval rating and impact factor inflation. The data are analysed for the most highly cited journals in economics, management, business and business–finance on the basis of 2YIF. In addition to evaluating research in the most highly cited journals in economics, management, business and business–finance, the paper evaluates alternative RAM, highlights similarities and differences in RAM criteria, finds that several RAM capture similar performance characteristics, and finds that immediacy and PI‐BETA are not highly correlated with other RAM. Harmonic mean rankings of the 12 RAM criteria are also presented. Emphasizing 2YIF to the exclusion of other useful RAM criteria can lead to a distorted evaluation of journal performance and influence.     

Journal ratings and footprints: a North American perspective of organizations and the natural environment journal quality

Organizations and Natural Environment (ONE) research has been published in mainstream management journals and an increasing array of specialty journals dedicated to ONE topics since the formation of the ONE interest group more than a decade ago. While the abundance of outlets for ONE research may be perceived as a positive development for the field, it also raises difficulties for establishing credibility for the relative quality of ONE scholarship. This study is the first of its kind to assess the journal quality of mainstream management and specialty journals that publish ONE‐related research. I triangulated methods by first utilizing a three‐stage process to survey senior ONE scholars and then utilized a footprint approach to rate 25 journals publishing ONE research. Results are compared with those of previous studies and are discussed in light of senior scholar commentary. Copyright © 2005 John Wiley & Sons, Ltd and ERP Environment.     

Applied econometrics rankings: 1989–1995

This paper ranks academic institutions by publication activity in applied econometrics over the period 1989–1995. Fourteen leading international journals that publish applied econometrics articles are used to provide the database. The rankings are based on standardized page counts of articles published in these journals over the stated period. A ‘Hall of Fame’ is developed listing the top 100 individual producers of applied econometrics in the fourteen journals considered. To control for quality differences among the applied journals, separate rankings are provided both for institutions and for individuals according to econometrics publications by journal. Copyright © 1999 John Wiley & Sons, Ltd.     

Ranking Economics Journals Using Data From a National Research Evaluation Exercise

This paper describes an algorithm for creating a ranking of economics journals, using data from the 2014 UK Research Excellence Framework (REF) exercise. The ranking generated by the algorithm can be viewed as a measure of the average quality of the papers published in the journal, as judged by the REF Economics and Econometrics sub‐panel, based on the outputs submitted to the REF.     

JMS at 50: Trends over Time

We present an analysis of the articles published in the Journal of Management Studies since its inception to assess to what extent JMS has: maintained its leading international ranking; maintained its founding mission as a broad based management journal; and remained a broad based management journal compared to other general management journals. In terms of its impact factor and citations despite reaching a low point in 2001, we find that JMS today ranks higher than it has ever done throughout its 50‐year history. From our content analysis covering the life‐cycle of JMS, we find four areas have been the most frequently represented, although their relative importance varies between decades: Organizational Management/Behaviour, Strategy, Human Resource Management, and General Management, accounting for 67 per cent of articles published over the period. JMS has strengthened its international author distribution through the increase in authors from the EU especially; the period 2000–04 which saw the predominance of UK authors was an anomaly. There are marked differences between type of article and author country of origin. Our comparative analysis of the word networks between the journals JMS, AMJ, ASQ, and HRM shows that over each decade the papers in the first three normally form a single cluster, indicating that the words used in the papers in the different journals are similar, while papers from HRM often form an outlining group. Notably, in the early 2000s papers in JMS form a distinct cluster, with papers from HRM paralleling the anomaly identified in the content analysis. Overall, JMS has regained its distinctiveness as a broad‐based international management journal, not favouring any particular theoretical or empirical approach.     

The performance of the global bottom-up approach in the M5 accuracy competition: A robustness check

The M5 accuracy competition has presented a large-scale hierarchical forecasting problem in a realistic grocery retail setting in order to evaluate an extended range of forecasting methods, particularly those adopting machine learning. The top ranking solutions adopted a global bottom-up approach, by which is meant using global forecasting methods to generate bottom level forecasts in the hierarchy and then using a bottom-up strategy to obtain coherent forecasts for aggregate levels. However, whether the observed superior performance of the global bottom-up approach is robust over various test periods or only an accidental result, is an important question for retail forecasting researchers and practitioners. We conduct experiments to explore the robustness of the global bottom-up approach, and make comments on the efforts made by the top-ranking teams to improve the core approach. We find that the top-ranking global bottom-up approaches lack robustness across time periods in the M5 data. This inconsistent performance makes the M5 final rankings somewhat of a lottery. In future forecasting competitions, we suggest the use of multiple rolling test sets to evaluate the forecasting performance in order to reward robustly performing forecasting methods, a much needed characteristic in any application.     

The relative importance of journals used in operations management research A citation analysis

Journal rankings are important for a variety of reasons, most importantly as the basis of academic tenure and promotion decisions. A common approach in other fields is the citations methodology which is used in this study to determine the relative importance of various operations management (OM) journals. The citations analysis noted the frequency that all other journals are cited in Decision Sciences, Journal of Operations Management, and Management Science during the period 1992 to 1994. Rankings are provided based on total citations, citations per article, and citations per words published. The study shows that the journals with the most importance to OM research are Management Science, Decisions Sciences and Operations Research. Other journals important to OM research are the Harvard Business Review, Journal of Operations Management, and IIE Transactions.     

Covariance estimation: do new methods outperform old ones?

This paper compares three new methods of estimating the asset returns covariance and evaluates their performances with the conventional covariance estimation methods. We find that taking a simple average of the historical sample covariance matrix and the covariance matrix estimated from the single-index model provides the best overall performance among all competing methods. In addition, we find that commonly used assessment criteria provide systematically different rankings, which explains the preferences to different types of estimation methods in the existing literature. We believe the difference between our results and those of previous studies may be partly due to the differences in the ratio of the time series observations to the number of stocks in the samples that have been used in different studies.     

Great Places to Work®: Resilience in Times of Crisis

We study the resilience of the “100 Best Companies to Work for in America” in times of financial crisis by analyzing their long‐term financial performance. Apart from implementing methods that tackle the statistical problems of stock returns, we use a conditional model to measure financial performance in periods of market growth (bull markets) and market downturn (bear markets). We find that best places to work are indeed resilient in times of crisis since neither their financial performance nor their systematic risk are affected during bear markets: top companies continue to outperform the market during periods of crisis, and the performance of lower‐ranked great workplaces does not deteriorate. Moreover, we find that previous studies were overestimating performance, and only great workplaces on the top half of the rankings exhibit positive excessive returns. © 2015 Wiley Periodicals, Inc.     

Different but Equal? On the Contribution of Dissident Economists

The availability of publication and citation databases facilitate construction of rankings of economics journals, economists, and departments. Mainstream economists typically find the research questions and methods of heterodox economists dubious, and this reaction creates a bias against heterodox research in peer review, which extends to traditional bibliometric rankings. University administrators interested in improving the quality of the economics department, as conventionally measured, will see heterodox economists as a liability. But establishing the bias of conventional metrics of scholarship quality and the consequences of these biases does not establish the comparable worth of the contributions of dissident economists. This special issue seeks innovated ways to document the relative quality of the contributions of dissident economists.     

Desingularization and Dequalification: A Foray Into Ranking Production and Utilization Processes

Abstract

Although some authors highlight the benefits of journal rankings, previous research is often highly critical of them, insinuating that they can lead to desingularization of academic journals (i.e. their impoverishment and standardization) and dequalification of researchers (i.e. a weakening of researchers’ ability to evaluate academic research). However, as very few authors have empirically assessed these presumptions, we aim to address this gap in the literature. Based on Lucien Karpik’s notions of singularities, judgment devices, forms of involvement, and emulation and rivalry, we assess whether the processes surrounding the production and use of journal rankings might lead to desingularization and dequalification. Our findings support previous research by highlighting that processes where passivity and heteronomy (i.e. lack of autonomy) prevail are conducive to desingularization, rivalry and dequalification. Our findings, however, introduce some nuances into the debate by underscoring instances where emulation logic is employed instead of mere rivalry logic, and where substantial judgment devices and active involvement are mobilized in the production and use of rankings, thereby somewhat alleviating the spread of desingularization and dequalification. Ultimately, our study raises questions that point to a need for serious collective reflection within the academic community on the processes by which published research is evaluated.     

《会计研究》三十年论文质量评价——基于《中国知网》引证文献的统计分析

通过对《会计研究》1980年创刊号始至2009年末期30年间发表的学术论文总体被引用状况的统计,描述了该刊三十年发表论文被引状况的分布及在不同时期的变化规律;通过对发表论文被引频率的排名及统计分析,获得在该刊发表的本学科优秀论文和活跃作者及任职机构;通过对论文作者单位的分析,发现该刊论文作者分布服从Lotka分布。结论表明,《会计研究》30年来发表的论文总体上反映了中国会计研究的历史发展,代表了各个年代中国会计研究的相应学术水平。     

高等医学院校医学期刊管理的探索

本文简述了医学期刊的发展,医学期刊的出版现状、前期的评价工具指标,医学期刊的网传方式及电子网络版生物期刊的服务方式。探究高等医学院校如何对医学期刊进行综合管理。为医学科研人员及医务工作者提供更有利的医学信息服务。