Article Text
PDF
Abstract
Introduction The analysis of surgical research using bibliometric measures has become increasingly prevalent. Absolute citation counts (CC) or indices are commonly used markers of research quality but may not adequately capture the most impactful research. A novel scoring system, the disruptive score (DS) has been found to identity academic work that either changes paradigms (disruptive (DIS) work) or entrenches ideas (developmental (DEV) work). We sought to analyze the most DIS and DEV versus most cited research in civilian trauma.
Methods The top papers by DS and by CC from trauma and surgery journals were identified via a professional literature search. The identified publications were then linked to the National Institutes of Health iCite tool to quantify total CC and related metrics. The top 100 DIS and DEV publications by DS were analyzed based on the area of focus, citation, and perceived clinical impact, and compared with the top 100 papers by CC.
Results 32 293 articles published between 1954 and 2014 were identified. The most common publication location of selected articles was published in Journal of Trauma (31%). Retrospective reviews (73%) were common in DIS (73%) and top CC (67%) papers, while DEV papers were frequently case reports (49%). Only 1 publication was identified in the top 100 DIS and top 100 CC lists. There was no significant correlation between CC and DS among the top 100 DIS papers (r=0.02; p=0.85), and only a weak correlation between CC and DS score (r=0.21; p<0.05) among the top 100 DEV papers.
Conclusion The disruption score identifies a unique subset of trauma academia. The most DIS trauma literature is highly distinct and has little overlap with top trauma publications identified by standard CC metrics, with no significant correlation between the CC and DS.
Level of evidence Level IV.
- research
- publishing
Data availability statement
Data are available in a public, open access repository. Data are available in a public, open access respository.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Bibliometric analysis is being used more frequently to evaluate academic production and relevance; however, most analytic techniques rely on biased indicators that may overlook historical and clinically relevant literature.
A novel bibliometric scoring system, the disruption score, has previously been used and validated to identify previously overlooked and distinct academic surgical literature that changed the course of surgical paradigm and influenced future research.
Until now, a thorough evaluation of the entirety of trauma surgery literature based on disruption score analysis has not previously been performed.
WHAT THIS STUDY ADDS
Disruption score analysis of trauma surgical literature identified a distinct set of historical and important previously overlooked publications.
Interestingly, papers that were deemed to disrupt the paradigm of trauma academia or reinforce novel ideas were not strongly correlated with citation count.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The results of the study allow for a proper appreciation of a more complete of trauma academic research.
The disruption score can help frame the proper historical context of important trauma work as it relates to previously shifting thought and help identify early work that shaped our current trauma management.
Introduction
Academic surgical research has seen exponential growth in output over the past few years.1 2 However, the increased academic production makes it difficult to maintain a pulse on the most impactful work and threatens to overwhelm the ability to maintain currency. Due to its theoretical ability to diminish noise and provide insight into grading relevant academic output,
bibliometric analysis has gained popularity in the evaluation of both researchers and the work performed in the surgical community. The popularity of bibliometrics has also grown to judge individual academic literary performance; the process of both the hiring and promoting academic surgeons has been found to been impacted by bibliometric scoring systems.1 3 Despite their impact on careers, these scoring systems are not without limitations.1 4–6 Common bibliometrics, such as citation count (CC) and the H-index, primarily derive their scale from a simple count of total citations of a published article. The flaws of these analytic frameworks are obvious: they do not factor the relevance or scientific impact of the citation and are biased against older publications and authors citing their own work.5 6 Moreover, they were not designed to capture academia that imparts major impact and change in a given area of research. Previous bibliometric analyses cannot judge which literature has changed the previously held dogma or paradigms—work that is truly ‘disruptive (DIS)’.7
These glaring deficiencies led to the development of a new metric called the disruption score.7 8 The disruption score, a ratio between –1 and 1, measures the degree to which a published work introduces a new idea compared with previous literature. Negative scores reflect developmental (DEV) work and identify publications that continue to expand on known ideas and further that current paradigm of scientific thought. Positive scores reflect DIS work and reflect literature that changes the known paradigm of cited related work, thus guiding the scientific thought in a new direction.
Disruption score analysis can identify important scientific work previously underappreciated by other bibliometric techniques, a strategy recently used to highlight overlooked research in multiple surgical subspecialties.2 9–16 DIS surgical work is not common; only 25% of the PubMed library publications are considered DIS, and only 10% have scores >0.10.11 Recently, we performed a bibliometric analysis of military-specific trauma publications using the disruption score.17 The results of that previous study identified a unique subset of relevant military-specific trauma literature previously overlooked by conventional metrics.17 However, this analysis has not specifically been applied across the plethora of trauma surgery academic offerings. In order to identify the most DIS trauma publications as well as previously overlooked works, the goal of this study was to identify the top 100 DIS and DEV trauma publications by disruption score and compare them with the top 100 publications using the standard CC metric. We hypothesize disruption score analysis will identify distinct academic trauma surgical work compared with publications identified by CC alone.
Methods
A PubMed search was performed in August 2022 in order to extract the PubMed identifiers (PMID) of all articles published in core trauma surgical journals from 1954 to 2014. Core trauma surgical journals were selected and vetted by the authors and librarian based on impact factor. These included medical journals with a specific focus on trauma that were actively publishing and searchable on PubMed. In order to capture relevant work published in non-trauma-specific journals, key trauma Medical Subject Headings terms were used to identify published work from previously peer-referenced top surgical academic journals9 during the same time period (online supplemental table 1). The PMIDs were merged with a validated dataset that contains the disruption scores on papers from 1954 to 2014.7
Supplemental material
The disruption score is defined by the calculation: ‘X=(A−B)/(A+B+C), where X=disruption score for paper X, A=number of future papers that cite paper X without paper X’s references being cited simultaneously, B=number of future papers that cite paper X and at least one of paper X’s references and C=number of future papers that cite one of the references of paper X, but not paper X itself”.11 Positive scores are considered DIS, with results closer to 1 being the most DIS; negative scores are classified as DEV, with scores the furthest from 0 to be the most DEV.
The top 100 DIS and DEV papers were identified by disruption score. The top 100 most cited (CC) papers were captured via the National Institutes of Health iCite tool.18 For the top 100 DIS papers, analysis was performed to identify journal location and research design. CC of the top DIS and top DEV papers were compared with the top CC papers using the Mann-Whitney U test. Correlation coefficients between disruption scores and CC were estimated. All variables were analyzed using IBM SPSS Statistics V.28 software (IBM, Armonk, New York, USA).
Results
Professional PubMed literary search identified 32 293 articles published between 1954 and 2014, as this is the same time period included in the disruption score database. Of all identified papers, 31% (n=10 022) were published in the Journal of Trauma; Injury was the second most common journal with 18.2% (n=5852). The average disruptive score (DS) across all papers was −0.002 with a median of 0 (IQR −008–0). Figure 1 shows the per cent of DIS and DEV papers from the 17 most commonly identified journals. Regarding CC, the average citations were 40.5.
Distribution of disruptive and developmental papers from the 17 most commonly identified journals.
The top 100 DIS and top 100 DEV papers are shown in online supplemental tables 2 and 3, respectively. The top DIS papers have disruption scores ranging from 0.16 to 0.76 and CC ranging from 0 to 754. The majority of the DIS papers were published in the Journal of Trauma and Acute Care Surgery (63%). The types of papers were most commonly retrospective reviews (73%) and commonly focused on injury management or quality improvement (figures 2 and 3). Of these, the most DIS paper was ‘Epidemiology of major trauma and trauma deaths in Los Angeles County’, published in a 1998 issue of the Journal of the American College of Surgeons. This paper, also the 15th most cited of the top DIS papers, was an epidemiological assessment of the trauma burden in one of the largest cities in the country. The most cited of the top DIS papers was ‘Organ injury scaling: spleen and liver (1994 revision)’ published in a 1995 issue of the Journal of Trauma. However, this was only the 31st most DIS paper, with a score of 0.26.
Distribution of research design of the top 100 disruptive trauma surgery articles.
Distribution of clinical focus of the top 100 disruptive trauma surgery articles.
The top DEV papers revealed a similar publication distribution, as 64% were published in the Journal of Trauma and Acute Care Surgery. In contrast to the top DIS papers, the top DEV papers were most commonly case reports (49%). The most DEV paper was ‘An overlapping pubic dislocation treated by closed reduction: case report and review of the literature’ published in a 1989 edition of the Journal of Surgery. This case review describes the repair and reconstruction of an overlapping pelvic injury without associated urogenital trauma. However, the paper was only the 64th most cited top DEV paper. The highest referenced top DEV papers was ‘Prospective study of blunt aortic injury: multicenter trial of the American Association for the Surgery of Trauma’ published in Journal of Trauma. Despite the high CC (495), this was only the 85th most DEV paper.
The most cited paper identified was ‘A national evaluation of the effect of trauma center care on mortality’, a 2006 review published in the New England Journal of Medicine. This paper identified a potential link between improved in-hospital mortality and designated trauma centers. While this paper has a positive DS (0.003), its distance from 1 precludes it from being labeled as highly DIS. On review, only 30% of the top CC papers have positive disruption scores and are considered DIS. The top CC papers were most commonly published in Journal of Trauma (39%).
The papers found on the top DIS, DEV, and CC are different from one another. Only one top DIS paper was found among the top 100 CC papers. Identifying papers by disruption yielded a distinct sample compared with highly cited papers, as evidenced by a lack of correlation between disruption metrics and CC (r=−0.02; p=0.85) (figure 4). Only two papers were found on both the top DEV and top CC lists. However, DEV scores (negative disruption scores) were only weakly correlated with CC (r=0.21; p=0.04) (figure 5).
Scatter plot of citation count by disruption scores of the top 100 most disruptive trauma papers.
Scatter plot of citation count by disruption scores of the top 100 most developmental trauma papers.
Discussion
While previous analysis of academic disruption in military-specific trauma publications has been performed, this is the first analytic use of the disruption score to be applied across the vastness of trauma literature. We identified the most DIS and DEV trauma papers and compared them to the highest cited papers of the same research field. This analysis revealed DIS papers and highly cited papers are distinct entities with no strong correlation. Additionally, DEV and highly cited trauma publications are not synonymous, as an article’s DEV score was only weakly correlated with CC.
Bibliometric analysis was originally developed to wade through the noise of academia and evaluate relevant literature.7 8 However, since some measures focus on the article while others highlight the author, proper identification and evaluation of academic publications has previously been disjointed. For example, author and institution effect is highlighted by the H-index and m-score.19 20 The ‘relative citation ratio’ (RCR) is an enhanced citation-based score that attempts to compare the importance of a publication against its field of academia.21 Despite its improvements over conventional bibliometrics, the RCR is not designed to explain an article’s influence in respective field. Even with these limitations, these scores are often touted as synonyms for academic achievement.3 19 The gaps in these previously overused scoring systems may be bridged by disruption analytics. The disruption score can identify unique academic literature previously overlooked by conventional means and qualify its influence on disrupting dogma or solidifying developing thought. It is worth nothing that this paper does not argue for one bibliometric analysis over another, but merely highlights important trauma work previously overlooked by other measures of relevance.
Disruption analytics have been applied across multiple surgical specialties with similar results presented here.2 9–17 Sullivan et al found multiple articles describing early clinical outcomes and important surgical innovations in pediatric surgery that were not among their highest referenced companions.11 In an evaluation of the colorectal surgical literature, Becerra et al identified the top DEV contributions to the colorectal field.2 As with the pediatric literature, the colorectal DIS and DEV scores did not correlate highly with CC.2 11
Perspective on the trajectory of trauma academia is gained by the evaluation of the top DIS and DEV articles. DIS papers, those that are often cited separately from their own referenced work, identify periods of paradigm shift and practice changes. The most DIS paper, ‘Epidemiology of major trauma and trauma deaths in Los Angeles County’ describes the climate of the traumatic burden affecting one of the largest cities in the world. This paper shed light on socioeconomic differences in trauma injury and care across Los Angeles. In today’s climate, the findings of this paper continue to ring true, as disparities in trauma care are still affected by socioeconomic status and access to care. The second most DIS paper, ‘Pediatric trauma. The no. 1 killer of children’, also holds weight today. This paper documented that change in causes of pediatric mortality. Before this paper, the top causes of pediatric death were medical diagnoses. This paper showcased the increasing pediatric trauma epidemic and its relationship to mortality. Nationally, this paper has become prophetic, as the burden of trauma, specifically gun violence, continues to plague our pediatric population. Each of the top two DIS papers were cited <100 times, but their impact is obvious and relevant. The use of citation-only metrics would overlook these seminal works that speak to the current national trauma culture.
DEV trauma papers appear to affirm ideas, surgical techniques, and practices—often those that later become solidified. For example, one of the most DEV paper, ‘Prospective screening for blunt cerebrovascular injuries: analysis of diagnostic modalities and outcomes’, described the prospective evaluation of early blunt cerebrovascular injuries (BCVI) and its role in stroke prevention. This paper helped pave the way for multiple works that influenced our current knowledge and implementation of BCVI screening protocols.
This study has multiple limitations that merit mention. Because the disruption score database only captured years 1954–2014, it is likely that some influential work has been missed in this analysis. However, these broad capture dates have been peer-used to evaluate other surgical subspecialties and we feel that 60 years encompasses a significant amount of trauma literature worth analyzing.2 9–17 Additionally, the strength of the disruption score is found in older articles, since it requires some amount of time to determine the presence or absence of citations. Another limitation is the amount of captured DIS literature considered that is not highly scientific research. Becerra et al, also finding this in their analysis of the colorectal literature, argued that while more rigorously designed research endeavors may gain more citation traction by building on literature that is difficult to supplant.2 Editorials, commentaries, and case reports may represent novel ideas supplanting previously cited literature. Additionally, not every research question can be answered using level 1 evidence found in a randomized controlled trial. DIS literature may point to the benefit of future work and help guide additional work and consensus guidelines. Despite this being a limitation of its sole method in identifying sentinel literature, we believe that it still yields important thoughts that would otherwise be overlooked. Although this disruption score has been validated to identify impactful achievements, its mathematical application to surgical literature has the potential to identify DIS papers without clinical impact.7 Papers with extremely low references may inflate the disruption score, especially if the paper is not highly referenced. Other advanced impact scoring systems, like RCR, may be better suited in that scenario. However, the addition of this score to other bibliometrics adds valuable insight into the clinical impact of an academic work.
Conclusion
Analysis of trauma literature using solely conventional bibliometric techniques misses important historical and educational literature. We used the disruption score to identify unique and important work that either enhanced or changed trauma care. This analysis, providing a unique historical assessment, enables previously overlooked trauma literature to be properly recognized and appreciated. Moving forward, the disruption score should be included in the bibliometric armamentarium of how trauma academia is evaluated.
Data availability statement
Data are available in a public, open access repository. Data are available in a public, open access respository.
Ethics statements
Patient consent for publication
Ethics approval
Given that this study did not involve human or animal data, local institutional board approval was not required.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors JD: contributed in literature search, study design, data collection, data analysis, data interpretation, writing, and critical revision. JW: study design, data collection, data interpretation, writing, and critical revision. EW: study design, data interpretation, writing, and critical revision. AG: contributed in study design, writing, and critical revision. BE: contributed in study design, writing, and critical revision. KM: contributed in study design, writing, and critical revision. MS: contributed in study design, writing, and critical revision. KI: contributed in study design, writing, and critical revision. MJM: contributed in literature search, study design, data collection, data analysis, data interpretation, writing, and critical revision. JD is the guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Disclaimer The views presented here are those of the authors and do not necessarily reflect those of the US government, the Department of Defense, or the United States Army.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.