Comparison of evaluation indexes for Gastroenterology and Hepatology journals in different databases

Jia-Yuan Li,Zhi-Han Yan,Ze Xiang, Ce Gao,Jian wu

Abstract

Key Words: Journal Article Influence Index; Journal Impact Factor; CiteScore; Gastroenterology and Hepatology; Scientometric index

lNTRODUCTlON

The quality assessment of peer-reviewed published research is important for the reputation, substance and growth of various professional associations, individual scientists, and academic institutions, as well as the funding organizations that evaluate and support them[1]. The quality of scientific contributions is primarily assessed on a temporal basis, with quantitative evaluation of the long-term impact in a field or discipline. The impact of an individual scientific article can be inferred from the citations that it receives.A similar principle is applied to evaluation of the journals that publish these scientific articles[2]. These long-standing efforts have led to researchers proposing various methods that improve the assessment of the quality of scientific journals[3,4]. What most of these methods have in common, though, is the use of complex mathematical algorithms to analyze networks of scientific papers to estimate citation quality.

First proposed by Eugene Garfield in 1955, the Scientific Citation Index, Journal Citation Reports[(JCR); published by the Institute for Scientific Information (ISI)] aims to rank, evaluate, classify, and compare journals[5]. The involved metrics are calculated based on the number of articles published by a journal and the number of times that a journal is cited. Moreover, they have been widely adopted as tools to evaluate researchers and research work in a wide range of scientific settings. One of the most prominent among such indicators is the Journal Impact Factor (JIF).

In addition to the JIF, other metrics provided by the ISI include total citation frequency, immediacy index, number of source entries published in the current year, frequency of citations in the previous 2 years, cited half-life, and the ratio of different citations for each article. The ISI introduced a simplified system in 1974, along with a list of topic categories and an accompanying catalog of the total 176 JCR journals. In recent decades, the different journal categories have been subjected to many holistic analyses. The resultant definitions of the common characteristics that underpin particular types of journals and relate to the JIF have served as a useful tool for researchers, both in the scientometric field and in general as contributing authors, to better evaluate journal impact[6,7]. To this day, journals are ranked by JIF within their assigned category. The journals listed in the JCR are further subgrouped by the ranking of JIF-related indicators (i.e.JIF variation coefficient,etc); this greater detailed categorization has allowed scholars to perlustrate the impact factor values more intuitively from a holistic and comprehensive perspective.

JIF has been the most widely used indicator of quality of scientific journals over the past decades[8].However, in accordance with the 1999 announcement by the ISI/JCR that the accuracy of JIF is not fully guaranteed[9], it is important to note that the methodological considerations in the JIF calculation still include a lack of assessment of the quality of citations, the inclusion of self-citations, poor comparability between different scientific fields, and an analysis of publications mainly in English[10]. This is in addition to the fact that JIFs of journals representing different disciplines are not comparable to each other.

On December 8, 2016, Scopus launched the CiteScore (CS) quality metric, in direct competition of JIF but which was developed specifically for journals indexed by Scopus. Over the past few years, the number of journals assigned a CS has increased dramatically, especially for journals that are not included in the JIF annual assignments. Scientometric studies evaluating the relationship between CS and JIF have revealed that although there is a strong correlation between the two metrics, there are also obvious complex differences[11,12]. While CS may be more balanced and most certainly is more transparent[13], it also shares some key limitations with the JIF[14,15].

Reference Citation Analysis(RCA) is a very recently launched open multidisciplinary citation analysis database based on artificial intelligence technology. This database covers a wide array of seemingly disparate disciplines such as business, economics and management, chemistry and materials science,engineering and computer science, health and medical sciences, humanities, literature and arts, life sciences and earth sciences, physics and mathematics, and social sciences. Users can search the collective literature based on fields such as author, category, DOI, ISSN, keyword, ORCID number, publication name, PubMed ID, and title to track original innovative research results and cutting-edge progress; they can also sort results by an article impact index metric. Importantly, the results analysis functionality culminates in a comprehensive and customizable report of the retrieved results.

Based on theRCAdatabase, theJournal Article Influence Index(JAII) metric is officially available as a new indicator of journal quality that is calculatedviathe normal approach of quantifiable citations.Systematically comparing this new metric to traditional journal evaluation metrics will help ensure the accuracy ofJAII. With acknowledgement of the continuous deepening of research in the field of Gastroenterology and Hepatology of recent years[16], we performed such a comparative analysis to determine the similarities and differences betweenJAIIand JIF/CS as journal evaluation indicators,with the ultimate aim of providing an intuitive method for visual representation of the related data.

MATERlALS AND METHODS

Data sources

The raw data for this study was obtained in July 2022 from the official websites of the institutions that released each metric under consideration. We searched the JCR to obtain the 2021 JIF list, downloaded the CS list updated in July from the Scopus website, and collected the 2022JAIIlist from theRCAdatabase (www.referencecitationanalysis.com). In addition, we also searched for information related to the characteristics of these scientific journal quality indexes for reference.

Besides, based on the results ofRCAsearch by the Gastroenterology and Hepatology category, we comparedJAIIto JIF and CS respectively. The resultant data from theRCAdatabase were used as the matching benchmark, and the matching method was based on ISSN, EISSN, and journal name.

RESULTS

Statistical analysis and visualization

The Gastroenterology and Hepatology-categorized journals identified in each database are presented in Table 1 (grouped by the evaluation indicator and in descending order according to the respective quality metric value). In total, 102 journals carried aJAII, 81 carried a JIF, and 76 carried a CS (all assigned in 2021).

Next, in order to make an intuitive comparison between the three evaluation indicators, we drew a scatter distribution plot for JIF-JAII(Figure 1A) and CS-JAII(Figure 1B), and plotted a single-timepoint uniform curve using the least squares method[17]. In this case, we took an intersection, considering that some journals withJAIIhave no JIF or CS. It can be seen from the figure that in the evaluation of lowerquality journals, the linearity ofJAIIand JIF/CS has greater overlap, but in the evaluation of higherquality journals, the randomness of the data is greater. Journals with a large deviation between JIF and

JAIIincludeNature Reviews Gastroenterology & Hepatology,Lancet Gastroenterology & Hepatology,Seminars in Liver Disease, and so on. Journals with a large deviation between CS andJAIIincludeGut,Journal of Hepatology, Gastroenterology, and so on.

The results of the combined analysis of the three journal evaluation indicators are visualized in Figure 2A-C[18]. Figure 2A gives a comparison of the values between the three evaluation indicators of the same journal (73 in total, taking the intersection). Figure 2B gives the JIF-JAIIratio and CS-JAIIratio for each journal. Figure 2C gives the values of JIF and CS in descendingJAIIorder.

Finally, we combined the three journal evaluation indicators together, and through a histogram(Figure 2D), we can more clearly see the impact of the joint evaluation of the three on the ranking of journals without weight. This can also be used as a reference evaluation method.

Table 1 Comparison of Journal Article Influence Index, Journal lmpact Factor, and CiteScore in decreasing order of Journal Article Influence Index values

International Journal of Hepatology 14.249-6.1 European Journal of Gastroenterology & Hepatology 14.2272.586-Therapeutic Advances in Gastroenterology 13.8234.8025.8 Journal of Neurogastroenterology and Motility 13.5944.7257.4 Pancreatology 13.4973.9775.8 Hepatology Research 13.3324.9427.8 Gut and Liver 13.1934.3216.6 Digestive Diseases 13.0813.4214.2 BMC Gastroenterology 12.9912.8473.3 Endoscopy 12.5419.77611 Journal of Crohn’s & Colitis 12.43210.02 Colorectal Disease 12.3413.9174.4 Liver Cancer 12.17412.4312.6 Digestive and Liver Disease: Official Journal of the Italian Society of Gastroenterology and the Italian Association for the Study of the Liver 12.0965.165-Diseases of the Esophagus: Official Journal of the International Society for Diseases of the Esophagus 11.9692.822-Current Opinion in Gastroenterology 11.9292.7414.9 World Journal of Gastrointestinal Oncology 11.5523.4043.6 United European Gastroenterology Journal 11.4536.8667.9 Clinical and Molecular Hepatology 11.2518.3378.9 Digestive Surgery 11.2262.4594.2 Expert Review of Gastroenterology & Hepatology 10.8854.095-World Journal of Gastrointestinal Endoscopy 10.598--World Journal of Gastrointestinal Surgery 10.5792.505-Clinical and Translational Gastroenterology 10.454.3965.2 Clinical and Experimental Gastroenterology 10.149-5 Gastroenterology Research and Practice 9.9021.9193.7 Journal of Digestive Diseases 9.3023.3664.2 Cellular and Molecular Gastroenterology and Hepatology 9.2778.797-Digestion 9.1893.6725.1 Clinics in Colon and Rectal Surgery 9.0592.4033.5 Techniques in Coloproctology 9.0563.6994.6 Journal of Gastric Cancer 9.0313.1974.4 Hepatic Medicine: Evidence and Research 8.847--Annals of Hepatology 8.7823.3884.7 JHEP Reports 8.6939.9178.1 BMJ Open Gastroenterology 7.884-3.5 Clinical Endoscopy 7.72-3.5 Intestinal Research 7.651-6 Canadian Journal of Gastroenterology & Hepatology 7.6152.605-Digestive Endoscopy 7.1116.3377.5 Hepatobiliary & Pancreatic Diseases International 7.0523.355-Esophagus: Official Journal of the Japan Esophageal Society 6.7753.671-

Endoscopy International Open 6.725--Gastroenterology Report 6.6854.0634.9 Clinics and Research in Hepatology and Gastroenterology 6.593.1893.1 Journal of Clinical and Experimental Hepatology 6.236-5.3 Saudi Journal of Gastroenterology 6.2053.2144.3 Hepatitis Monthly 6.0371.2141.1 Hepatology Communications 6.0065.7017.7 Liver Research 5.941-6.3 Endoscopic Ultrasound 5.9325.2755.9 Gastrointestinal Tumors 5.556--Indian Journal of Gastroenterology: Official Journal of the Indian Society of Gastroenterology 5.311--Frontline Gastroenterology 4.933-3.8 Journal of Clinical and Translational Hepatology 4.5625.0656.4 Inflammatory Intestinal Diseases 4.474-0.2 Annals of Gastroenterological Surgery 4.4273.5835.5 Case Reports in Gastroenterology 4.117-1 Annals of Coloproctology 3.946-2.4 Translational Gastroenterology and Hepatology 3.945-5.5 Clinical Liver Disease 3.934-2.4 Journal of Gastrointestinal Oncology 3.0292.5873.3 CS: CiteScore; JAII: Journal Article Influence Index; JIF: Journal Impact Factor. “-” denotes lack of score assigned by the corresponding institution/database.

Figure 1 Scatter distribution plots for Journal lmpact Factor-Journal Article Influence Index and CiteScore-Journal Article Influence Index.

DlSCUSSlON

Comparison of databases and calculation principles

JIF:JIFs are obtained through the Web of Knowledge database using the Science Edition of JCR which collects citation data from more than 7300 science and technology journals worldwide. The IF of a Tyear journal is defined as the number of times that the journal has been cited in years T-1 and T-2 divided by the number of documents that can be cited in the journal in years T-1 and T-2[19].

CS:CSs are calculated using data from the Scopus database. CS has a publication window of 3 years before the 1-year reference window and counts the references from one document type to another[20].In other words, CS calculates the average number of citations of papers published in a journal for 3 consecutive years in the 4thyear. In a given year, CS is calculated as the amount of times that documents published in the previous 3 years were cited in that year divided by the number of documents published in those 3 years that were included in the Scopus database.

Figure 2 Visualization of the three journal evaluation indicators. A: Comparison of the values obtained upon application of the three evaluation indicators;B: Journal Impact Factor (JIF)-Journal Article Influence Index (JAII) and CiteScore (CS)-JAII ratios for each journal; C: Values of JIF and CS in descending JAII order;D: Histogram combining the three journal evaluation indicators together. JIF: Journal Impact Factor; CS: CiteScore; JAII: Journal Article Influence Index.

JAII:JAIIs, calculated as total citations divided by total articles, are based on journals and their citations included in theRCAdatabase.

Advantages and disadvantages of JAII

It is undeniable that theJAIImetric has its merits as a journal evaluation indicator. (1) Compared with JIF and CS,JAIIis able to break through the time limit disadvantage of the first two. Journals do not need to meet the waiting-time thresholds of JIF and CS to be accurately evaluated. As such,JAIIis able to evaluate more journals accurately in a near-real time manner, which explains why there are more journals with aJAIIthan those with a JIF/CS. (2) Compared with JIF and CS,JAIIis useful for assessing the performance of journals immediately upon its creation. Moreover, since a small number of articles in journals will result in a high JIF and CS at a given time,JAIIrelieves the chance of biased evaluation of journals.JAIIis also more conducive to a comprehensive assessment of the quality and performance of journals. And (3), compared with JIF and CS,JAIIis more conducive to high-quality journal evaluation.In addition to these advantages,JAIIhas a high degree of compliance with JIF and CS in the evaluation of journals with a lower impact.

Another important feature related to theJAIIis that theRCAdatabase, upon which it is based, can enable queries to journals by category, such as focused query of Gastroenterology and Hepatology,representing a ready convenience to researchers.

There exist disadvantages in theJAII. These include the lack of statistical timeliness, a feature by whichJAIIis slightly inferior to JIF and CS, and the lack of consideration to different developments of the same journal in different periods.JAIIalso shares some of the drawbacks of both JIF and CS, such as the lack of evaluation of citation quality and the inclusion of self-citations.

Non-linearity interpretation

As we have shown in Figure 1 and described textually in the “Results” section above, the linearity ofJAII-JIF-CS was clear for lower-quality journals but failed to match each other perfectly for higherquality journals.Our explanation is that JIF and CS are subject to changes in citation frequency and number of published articles in different years, and their correlation with time exacerbates the influence of human manipulability[21].JAIIreduces this time randomness. In addition, the JIF and CS of high-quality journals may be more susceptible to this effect, and their fluctuations can be effectively explained.

Threats to validity

In addition to the lack of evaluation of citation quality and self-citation, other factors may threaten the effectiveness of the evaluation parameters in use. Research on JIF, CS and other statistical standards for journal quality has shown that there are still many statistical violations at play, including those related to and arising from reliability, incomplete reporting of validity, insignificant results, insignificant effect sizes, and hypothesis checking, as well as uncorrected inferences and multiple comparisons from descriptive statistics[22].JAIIis also inevitably affected by the same, to at least some extent, and this limitation cannot be ignored.

CONCLUSlON

The main differences betweenJAIIand JIF/CS come from the differences in the scientific databases used as the cited sources, as well as the differences in the evaluation methods underpinning each of these indicators. Due to the JIF/CS time factor limitation, theJAIImethod based on theRCAdatabase is able to evaluate more journals. Besides,JAIIprovides more focused quantitative insight by considering categories of journal papers. In terms of practicality, the novelty introduced by theJAIIindicator is its open-accessibility to users (as opposed to a subscription service to select users). To summarize,JAIIis a reliable index to evaluate the quality of journals in near-real time.

In the future, scientometric researchers can focus on the differences of the different journal evaluation indexes to aid in their studies on the origin of nonlinear characteristics in order to put forward a more perfect journal evaluation standard. Meanwhile, researchers in general can exploit the distinct advantages of each as they currently stand to better understand journal quality and promote the impact of their own scientific communications.

ARTlCLE HlGHLlGHTS

Research perspectives

In the future, we hope to better explain the current existent nonlinear relationship among the three evaluation indexes, and combine a variety of journal evaluation indicators to allow for more comprehensive evaluation of journal quality by scientometric-focused and general researchers.

FOOTNOTES

Author contributions:Wu J designed the study and revised the manuscript; Li JY and Yan ZH performed the data analysis and manuscript drafting; Xiang Z and Gao C searched the literature and collected the data; Li JY wrote the paper; Wu J reviewed the results and made critical comments on the manuscript; All authors reviewed and approved the final version; Li JY and Yan ZH contributed equally to this work.

Supported bythe Youth Medical Talent of Jiangsu Province, No. QNRC2016475.

Conflict-of-interest statement:The authors declare that there are no competing interests associated with this manuscript.

PRlSMA 2009 Checklist statement:The authors have read the PRISMA 2009 Checklist, and the manuscript was prepared and revised according to the PRISMA 2009 Checklist.

Open-Access:This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BYNC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is noncommercial. See: https://creativecommons.org/Licenses/by-nc/4.0/

Country/Territory of origin:China

ORClD number:Jian Wu 0000-0003-0087-3744.

S-Editor:Chen YL

L-Editor:Wang TQ

P-Editor:Chen YL