Predictors of citations and altmetric scores in general surgery literature

Abstract

Objective: This study aimed to determine various article characteristics influencing the citations and altmetric scores using papers published in a year in four high-ranking surgery journals.

Material and Methods: We included all papers (n= 819 articles) published between January 2015 to December 2015 in the Annals of Surgery, British Journal of Surgery, JAMA Surgery and Journal of American College of Surgeons. Article characteristics were manually extracted. We determined citation count using the Web of Science database and used univariate analysis and negative binomial regression to determine which article characteristics affect citations and altmetric scores.

Results: Mean number of citations and altmetric score received by the article were 44.6 (0-475) and 19.2 (0-665) respectively. Majority of the articles contained at least one citation (98.3%) and altmetric score (98.2%). In regression analysis, citation count was significantly associated with the journal [Annals of Surgery (IRR= 1.93), JAMA surgery (IRR= 1.76)] and non-funded research (IRR= 0.83). The altmetric score was significantly associated with the country of the corresponding author (US) (IRR= 1.3), study subtopic, journal [JAMA surgery (IRR= 2.33)], non-funded (IRR= 0.74) and non-open-access publication (IRR= 0.44).

Conclusion: Article metrics were found to be associated with specific study subtopics, country of the corresponding author, funding, open-access publication and the journal. These results might help editors, reviewers and authors to produce, review and publish more impactful studies. A similar study in the future may help to better understand the changing dynamics of academic publishing.

Keywords: Citation count, altmetric attention score, bibliometrics, general surgery, negative binomial regression

Introduction

Medical research is the cornerstone of furthering medical knowledge, discovering new treatments, and improving the lives of patients, and requires a significant effort and expenditure of resources on the part of academic institutions, as well as their faculty and staff. While all research contributes to increasing the span of knowledge in medical science, quantifying the impact of an individual study is often a challenging task, which is traditionally done using citation count. The number of citations received by an article is often a surrogate of that article’s scientific impact and importance (1). With the advent of open-access publishing and all the literature being made publicly available, a new metric called “Altmetrics” became popular to measure the “social” impact of the study. This could be seen as a surrogate for social media attention, dissemination and influence (2). Altmetric score is complementary to the citation count and both correlate well with each other (3).

Previous studies have tried to evaluate article characteristics associated with higher citation count in surgical subspecialties such as neurosurgery, urology, orthopaedics and plastic surgery (4-7). However, little is known about the factors predicting higher citation counts in general surgery. A study conducted by Mullins et al. has shown that manuscripts with higher citation counts in the general surgery literature are more likely to be clinical, collaborative, multi-institutional, and larger and more sample size (8). However, the sample size is smaller and prone to selection bias. In addition, to the best of our knowledge, article characteristics on which the altmetric score depend have not been evaluated in the general surgical literature.

This study aimed to determine various article characteristics which can influence citation count and altmetric attention score using papers published in a year in the four high-ranking general surgery journals. The results might help editors, reviewers and authors to produce, review and publish more impactful studies which may be important to the field.

Material and Methods

Study Design and Data Collection

As the study did not involve human participants, research ethics committee approval was not applicable. We selected four top-ranked journals in “General Surgery” with high impact factors (IF) - The Annals of Surgery (AS) (IF 13.79), British Journal of Surgery (BJS) (IF 6.939), JAMA Surgery (IF 16.68) and Journal of American College of Surgeons (JACS) (6.532). These journals were chosen based on their importance in surgery according to the SCImago Journal Rank Indicator. We identified all research articles published in these journals during the one year between January 2015 to December 2015. Article types such as editorials and letters to editors were excluded, which yielded a total of 819 eligible studies.

In February 2022, after six hours of training in the use of the Web of Science (WoS) at the institute’s library, the authors extracted the following article characteristics from each of the above-selected articles and entered manually in a spreadsheet software. Data on citation count and altmetric attention score were entered between February 6th to February 14^th, 2022. We randomly selected 40 articles to confirm the accuracy of data collection.

The following data were extracted from each study:

1. Journal of publication (AS, BJS, JAMA Surgery, JACS)

2. Title of the study

3. Study design (experimental study, interventional, prospective cohort, retrospective cohort/case-control, cross-sectional study, case report/case series, systematic review/ meta-analyses and guidelines)

4. Study subtopic

5. Number of characters in the title (excluding trailing or double spaces)

6. The month of publication (from January 2015 to December 2015)

7. Page count

8. Country of the corresponding author

9. Number of authors

10. Number of references

11. Whether any funding was received (Yes/No)

12. Whether the paper was Open Access (Yes/No). Dependent variables:

13. Number of citations in WoS core collection

14. Altmetric attention score.

Statistical Analysis

We inserted data into a statistical database (SPSS, v.26.0, ©IBM Inc.) for analysis. Descriptive analysis was performed to quantitatively describe the features of the sample. Continuous variables were compared by independent t-test and analysis of variance (for normal distribution) for two or more than two groups respectively. Wilcoxon-rank sum test and Kruskal Wallis test were used for non-normal data - if the Shapiro-Wilk test was significant.

Regression: The mean of the number of citations, citations/year and the altmetric attention score was lower than their variance (overdispersion). Thus, the negative binomial regression model was used to determine the article characteristics which can affect the dependent variables. To transform the percentage of author self-citations into count data, we rounded off the percentage to the nearest whole number.

The analysis was performed on the entire study cohort. There were no missing values in the data set. All statistical tests were two-sided, and p values of less than 0.05 were considered to indicate statistical significance.

Results

Article Characteristics

A total of 819 articles were analyzed. Characteristics of the included articles are shown in Table 1. Most of the studies belonged to the category of gastrointestinal surgery (54.0%). “Prospective cohort” was the most common study design (29.2%) closely followed by the ‘retrospective cohort/ case control’ study (28.7%). Approximately a third of the studies were published in the AS (34.7%), and around half of the lead authors of the studies belonged to the United States (US) (48.4%). Most of the studies were not funded (57.1%) and were not open access (65.9%). Mean number of authors was 10 (5.7), mean character count in the title was 108.1 (35.5), mean page count was 8.1 (2.2), and mean reference count was 32.7 (17.7).

Mean number of citations received by the article in the WoS core collection were 44.6 (range 0-475). Mean altmetric attention score was 19.2 (0-665). The overwhelming majority of articles contained at least one citation (98.3%) and altmetric attention score (98.2%) respectively.

Univariate Analysis

In univariate analysis, study design, study subtopic, journal of publication, funded research, character count, reference count and the number of authors were statistically significantly associated with the number of citations. The altmetric attention score was statistically significantly associated with the study subtopic, the journal of publication, the country of the corresponding author, funded research and if the paper was open access (Table 2).

Regression Analysis

The variance inflation factor for each variable included in the regression model was <5. Thus, there was no multicollinearity and all the statistically significant variables in univariate analysis were put in the regression model. The results are shown in Table 3. Adjusted analysis showed that the study design was not statistically significantly associated with the citation count. Under the study subtopic, endocrine surgery, cardiothoracic surgery and plastic surgery were negatively associated with the altmetric score [incidence rate ratio (IRR= 0.21, 0.29, 0.19 respectively)]. “AS” was statistically significantly associated with the number of citations (IRR= 1.93) and the “JAMA surgery” was statistically significantly associated with both the number of citations and the altmetric attention score (IRR= 1.76, 2.33 respectively).

The altmetric score was statistically significantly higher when the corresponding author belonged to the US (IRR= 1.3). Nonfunded research was negatively associated with both citation counts and altmetric attention scores (IRR= 0.83, 0.74 respectively). Non-open access publication was negatively associated with the altmetric attention score (IRR= 0.44). Page count and the number of references were both borderlines positively associated with the number of citations.

Discussion

In this study, we evaluated what article characteristics could affect citation rates and altmetric attention score in highranked general surgery literature. We found that the study design was not associated with higher citation counts or altmetric attention scores. Earlier studies evaluating the effect of study design on citation rates have been mixed. While some studies based on orthopaedics or urology indicate that randomized trials and systematic reviews are associated with higher citation counts, other studies based on plastic surgery and emergency medicine fail to do so (5,7,9,10). However, the literature on the effect of study design on altmetric scores is sparse. Puzas et al. have evaluated 840 articles in dermatology and found that journal type, presence of conflicts of interest and open-access articles are associated with higher altmetric attention scores (11).

Surgical sub-fields such as endocrine surgery, cardiothoracic, vascular surgery and burns, plastic surgery were found to be significantly negatively associated with altmetric scores when compared to orthopaedic surgery. This could be due to a higher prevalence of certain diseases and/or more familiarity with particular fields to the people. Publication in AS was significantly associated with higher citation rates, and publication in JAMA Surgery was strongly associated with both higher citation rates and altmetric scores. This is expected given the fact that JAMA surgery has the highest impact factor among all surgical journals in the world (12). IF is a measure of the average number of citations received by the article in time. It is intuitive, therefore, that IF correlates well with citation counts. Moreover, there is enough evidence that across medical and surgical specialities, altmetric scores correlate well with citation counts (3).

US authors emerged as a significant predictor of higher altmetric scores in adjusted analysis. The reasons for this observation could be many. The US currently leads the world’s medical research output (13,14). This is largely in part due to high research funding and a diverse scientific community. In addition, most of the top-ranked journals have emerged from the US. American authors and reviewers are biased towards the articles published locally which can falsely amplify article metrics (15,16). Moreover, studies indicate that even among top journals, papers published by high-income countries gather more metrics than low and middle-income countries (17).

Funded research also emerged as a significant predictor of higher citation counts and altmetric scores. Non-funded articles roughly received 20% fewer citations and had 25% lower altmetric scores than the funded articles independent of the study subtopic and study design in adjusted analysis. A study conducted by Mosleh et al. shows that funded projects receive more citations than non-funded ones across life sciences (18). This indicates that funded projects have more visibility on social media and in the research community and are seen as a marker of quality. Thus, funding agencies should judiciously use resources to allocate funds to impactful studies. We also found that open-access publications have significantly higher altmetric scores than non-open access publications by nearly 2.2 folds. However, similar results were not obtained with citation counts. This has been seen in studies conducted in orthopaedic and otolaryngology literature as well (19,20). With the advent and popularity of “Scihub” and “Libgen” however, which provides free access to nearly 85% of the paywalled literature, the results could be significantly distorted as the majority of the medical literature may be ‘open access’ (21).

We used a robust methodology in this study. We used a large sample size and included both the traditional measure of scientific impact i.e. citation counts and more contemporary social media indicators of scientific impact i.e. altmetric attention scores to determine which article characteristics can affect these measures. The results could be helpful to journal editors, reviewers and authors to predict and invest time and resources to the papers which may have high impact in the field. We chose four journals with the highest impact factors. Since these journals are widely read by both the “experts” and the public, the results are more generalizable to the general surgery literature. We used a window of seven years postpublication to adequately capture the number of citations and altmetric scores.

This study has limitations. We used a retrospective observational design which can potentially generate confounding. In addition, there is a recent trend towards more collaborative research and the increased publication of systematic reviews and randomized trials and a relatively lower preference towards qualitative studies (22,23). Thus, the results might differ if the study is done in the future. It will be useful, therefore, to perform a similar study at a future time point to better understand the changing dynamics of the academic publishing. Similarly, the inclusion of smaller or more specialized journals may alter the results. However, we believe that the chosen journals have a broad readership and widely cover the subspeciality subtopics. We used WoS core collection for the citations. The results may vary if other databases are used including Google Scholar or Scopus which have been shown to have higher citation counts than WoS (24).

Conclusion

Article metrics were improved with specific study subtopics, when the author belonged to the US, when the research was funded, open access or published in a journal with higher impact factor. The results might help editors, reviewers and authors to produce, review and publish more impactful studies. Similar studies at a future time point will help to better understand the changing dynamics of the academic publishing.

Cite this article as: Chaudhary D, Acharya S, Aggarwal V, Huzaifa M, Kain P, Garg R, et al. Predictors of citations and altmetric scores in general surgery literature. Turk J Surg 2024; 40 (2): 145-153.

Externally peer-reviewed.

Concept - VA, DC, SA; Design - VA, DC, SA; Supervision - VA; Data Collection and/or Processing - All of authors; Analysis and/ or Interpretation - DC, SA, VA; Literature Search - DC, SA, VA, PK, KM, SK, AV; Writing Manuscript - DC, SA, VA; Critical Reviews - VA, RG, MH.

The authors have no conflicts of interest to declare.

The authors declared that this study has received no financial support.

References

1. Kelly CD, Jennions MD. The h index and career assessment by numbers. Trends Ecol Evol 2006; 21(4): 167-70. https://doi.org/10.1016/j. tree.2006.01.005
2. Altmetric. What are altmetrics? Accessed date: January 20, 2023. Available from: https://www.altmetric.com/about-altmetrics/whatare-altmetrics/
3. Kolahi J, Khazaei S, Iranmanesh P, Kim J, Bang H, Khademi A. Metaanalysis of correlations between altmetric attention score and citations in health sciences. BioMed Res Int 2021; 2021: 6680764. https:// doi.org/10.1155/2021/6680764
4. Lee KE, McMullen N, Kota H, Peterson K, Oravec C, Frey C, et al. Predictors of citations in neurosurgical research: A 5-year follow-up. World Neurosurg 2021; 153: e66-e75. https://doi.org/10.1016/j. wneu.2021.06.029
5. Willis DL, Bahler CD, Neuberger MM, Dahm P. Predictors of citations in the urological literature. BJU Int 2011; 107(12): 1876-80. https://doi. org/10.1111/j.1464-410X.2010.10028.x
6. Movassagi K, Kunze KN, Beck EC, Fu MC, Nho SJ. Predictors of 5-year citation rate in the orthopaedic sports medicine literature. Am J Sports Med 2019; 47(1): 206-11. https://doi.org/10.1177/0363546518810504
7. Lopez J, Calotta N, Doshi A, Soni A, Milton J, May JW Jr, et al. Citation rate predictors in the plastic surgery literature. J Surg Educ 2017; 74(2): 191-8. https://doi.org/10.1016/j.jsurg.2016.08.005
8. Mullins CH, Boyd CJ, Lindeman B. Factors associated with the highest and lowest cited research articles in general surgery journals. J Surg Res 2020; 250: 39-44. https://doi.org/10.1016/j.jss.2019.12.014
9. Bhandari M, Busse J, Devereaux PJ, Montori VM, Swiontkowski M, Tornetta Lii P, et al. Factors associated with citation rates in the orthopedic literature. Can J Surg 2007; 50(2): 119-23.
10. Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 2002; 287(21): 2847-50. https:// doi.org/10.1001/jama.287.21.2847
11. Iglesias-Puzas Á, Conde-Taboada A, Aranegui-Arteaga B, López-Bran E. Factors associated with high altmetric attention Score in dermatology research. Australas J Dermatol 2021; 62(3): e380-5. https://doi. org/10.1111/ajd.13591
12. For Authors | JAMA Surgery | JAMA Network. Accessed date: January 19, 2023. Available from: https://jamanetwork.com/journals/jamasurgery/pages/for-authors
13. Soteriades ES, Falagas ME. Comparison of amount of biomedical research originating from the European Union and the United States. BMJ 2005; 331(7510): 192-4. https://doi.org/10.1136/ bmj.331.7510.192
14. Soteriades ES, Rosmarakis ES, Paraschakis K, Falagas ME. Research contribution of different world regions in the top 50 biomedical journals (1995-2002). FASEB J 2006; 20(1): 29-34. https://doi.org/10.1096/ fj.05-4711lsf
15. Campbell FM. National bias: A comparison of citation practices by health professionals. Bull Med Libr Assoc 1990; 78(4): 376-82.
16. Link AM. US and non-US submissions: An analysis of reviewer bias. JAMA 1998; 280(3): 246-7. https://doi.org/10.1001/jama.280.3.246
17. Akre O, Barone-Adesi F, Pettersson A, Pearce N, Merletti F, Richiardi L. Differences in citation rates by country of origin for papers published in top-ranked medical journals: Do they reflect inequalities in access to publication? J Epidemiol Community Health 2011; 65(2): 119-23. https://doi.org/10.1136/jech.2009.088690
18. Mosleh M, Roshani S, Coccia M. Scientific laws of research funding to support citations and diffusion of knowledge in life science. Scientometrics 2022; 127(4): 1931-51. https://doi.org/10.1007/s11192- 022-04300-1
19. Zhang D, Blazar P, Kilgallen EE, Earp BE. Impact of conventional and open access publications in orthopaedic surgery. J Am Acad Orthop Surg 2021; 29(23): e1239-45. https://doi.org/10.5435/ JAAOS-D-20-01074
20. Crossley JR, Almasri M, Samaha N, Deklotz TR, Harley EH, Davidson BJ, et al. Citations and author characteristics in open-access and subscription-based otolaryngology journals. The Laryngoscope 2023; 133(1): 79-82. https://doi.org/10.1002/lary.30167
21. Himmelstein DS, Romero AR, Levernier JG, Munro TA, McLaughlin SR, Greshake Tzovaras B, et al. Sci-Hub provides access to nearly all scholarly literature. eLife 2018; 7: e32822. https://doi.org/10.7554/ eLife.32822
22. Fontelo P, Liu F. A review of recent publication trends from top publishing countries. Syst Rev 2018; 7(1): 147. https://doi.org/10.1186/ s13643-018-0819-1
23. Izwan S, Chan E, Ibraheem C, Bhagwat G, Parker D. Trends in publication of general surgery research in Australia, 2000-2020. ANZ J Surg 2022; 92(4): 718-22. https://doi.org/10.1111/ans.17543
24. Anker MS, Hadzibegovic S, Lena A, Haverkamp W. The difference in referencing in Web of Science, Scopus, and Google Scholar. ESC Heart Fail 2019; 6(6): 1291-312. https://doi.org/10.1002/ehf2.12583