You are here

Article Text

Article menu

Download PDFPDF

XML
Current opinion
Collecting data on organizational structures of trauma centers: the CAFE web service
  1. Mathias Brochhausen1,
  2. Jane W Ball2,
  3. Nels D Sanddal3,
  4. Jimm Dodd2,
  5. Naomi Braun1,
  6. Sarah Bost1,
  7. Joseph Utecht4,
  8. Robert J Winchell5,
  9. Kevin W Sexton6
  1. 1Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, Florida, USA
  2. 2American College of Surgeons, Chicago, Illinois, USA
  3. 3systEMS, Bozeman, Montana, USA
  4. 4Biomedical Informatics, University of Arkansas for Medical Sciences, Little Rock, Arkansas, USA
  5. 5Surgery, Weill Cornell Medical College, New York, New York, USA
  6. 6Surgery, UAMS, Little Rock, Arkansas, USA
  1. Correspondence to Dr Mathias Brochhausen; mbrochhausen{at}ufl.edu

Abstract

Background During the past several decades, the American College of Surgeons has led efforts to standardize trauma care through their trauma center verification process and Trauma Quality Improvement Program. Despite these endeavors, great variability remains among trauma centers functioning at the same level. Little research has been conducted on the correlation between trauma center organizational structure and patient outcomes. We are attempting to close this knowledge gap with the Comparative Assessment Framework for Environments of Trauma Care (CAFE) project.

Methods Our first action was to establish a shared terminology that we then used to build the Ontology of Organizational Structures of Trauma centers and Trauma systems (OOSTT). OOSTT underpins the web-based CAFE questionnaire that collects detailed information on the particular organizational attributes of trauma centers and trauma systems. This tool allows users to compare their organizations to an aggregate of other organizations of the same type, while collecting their data.

Results In collaboration with the American College of Surgeons Committee on Trauma, we tested the system by entering data from three trauma centers and four trauma systems. We also tested retrieval of answers to competency questions.

Discussion The data we gather will be made available to public health and implementation science researchers using visualizations. In the next phase of our project, we plan to link the gathered data about trauma center attributes to clinical outcomes.

  • United States
  • delivery of health care
  • health care economics and organizations
  • surveys and questionnaires
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/tsaco-2020-000473

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Beginning in the 1970s, the American College of Surgeons (ACS) worked to establish and refine criteria that set standards for trauma center organization, professional care provider qualifications, and performance.1 Until the last several years, these standards focused largely on measures of structure and process because only recently has it become possible to obtain reliable outcome data. The most recent ACS standards for trauma center verification were published in Resources for the Optimal Care of the Injured Patient in 2014.1 In 2008, the ACS established the Trauma Quality Improvement Program (TQIP) to build a process for trauma centers to measure and compare their risk-adjusted patient outcomes to similar organizations.2 Though TQIP has evolved and matured during the last decade, at this time, the correlation between organizational structure and patient outcomes is not yet defined.3–6 Further, beyond the standards themselves, the ways in which the particular attributes of trauma centers, and, on a larger scale, of regional trauma systems, contribute to optimal patient outcomes have not yet been identified or measured.6 7

    The TQIP efforts and the ACS trauma center verification standards paved the way for more rigorous research on the implementation of trauma centers and quality control. However, a gap remains between linking specific organizational structures to specific patient outcomes. Combining TQIP data with participation in a quality collaborative leads to significant improvement in patient outcomes compared with benchmarking alone.2 This suggests that only when institutional practice and resources are compared do centers obtain the maximum benefit from risk-adjusted data. As a result, institutional benchmarking alone does not identify the procedural or structural changes that are most likely to positively affect patient outcomes.

    A major gap is created by the lack of a controlled vocabulary and well-developed methodologies to collect information about trauma center attributes that contribute to better patient outcomes. This gap contributes to the fact that great variability in risk-adjusted mortality rates exists even among trauma centers functioning at a similar level.8–10 In designated level 1 and 2 trauma centers in a single state, the variability in the risk-adjusted OR of survival for each trauma center when compared with the best trauma center ranged from 0.2 (significantly different) to 0.9 (not significantly different).11 Even while comparing only level 1 trauma programs, the variability ranged from 0.4 (significantly worse) to 0.9 (no difference), suggesting that a significant difference in mortality is present. In Ohio, mortality varied from 3.8% to 24.2% across level 1 trauma centers despite similar patient characteristics and injury severity.12 Nationally, the median incidence of survival after trauma ranges from 52.6% to 87.3% depending on the region of the country where injured.13 These data support a wide variation of survival after trauma.

    Consider a hospital wanting to achieve better outcomes for injured patients in a developing nation with limited resources. It might be possible for the hospital’s administrative team to identify several trauma centers that have reported improving their patient outcomes as part of the TQIP effort. Achieving high-performance measures is an important goal, but the developing hospital cannot determine which trauma center attributes support the attainment of the high-performance standard.2 If the evolving institution has to choose between a major equipment purchase, operating suite upgrades, or changes in staffing patterns for physicians and nurses, to name a few, which choice might lead to the greatest return on investment? The same questions apply to remote and rural areas of North America and to more urban environments where some proliferation and duplication of trauma care resources have been questioned.6 14–16

    Comparative Assessment Framework for Environments of Trauma Care (CAFE) is a National Institutes of Health-funded project (R01GM111324) that aims to build a web-based system that collects detailed information on the particular organizational attributes of trauma centers and trauma systems to conduct an anonymous self-assessment of the organizational structures of such institutions. The future plan is to link the gathered data about trauma center attributes to clinical outcomes. Once this is complete, it will be possible to identify attributes of trauma centers and trauma systems that strongly correlate to patient outcomes.17

    One CAFE use case is to enable the comparison of the organizational structures in one evolving institution aiming for trauma center verification with the organizational structure of verified trauma centers. To facilitate that, CAFE provides a common terminology covering all relevant aspects of trauma center and trauma system management.18

    The purpose of this article is to introduce the trauma surgery community and trauma system community to our work conducted in collaboration with the ACS. We give an overview of the CAFE project, describe the methodologies underlying our data collection tools and the CAFE questionnaires, and describe the real-time comparison functionalities. The CAFE web services are open and freely available to the entire trauma care community both within the USA and internationally. In addition, this article reports on the outcome of the initial data collection and beta testing which led to significant improvement of the CAFE questionnaires.

    Methods

    The CAFE web-based infrastructure serves two main functionalities (figure 1):

    1. Allow representatives of trauma centers and trauma systems to enter information about their institution and receive a real-time comparison with the organizational structure of other institutions of the same type.

    2. Provide public health and implementation science researchers with data on organizational structures of trauma centers and trauma systems.

    Figure 1
    Figure 1

    The CAFE web service. Representatives of trauma centers or trauma systems use the questionnaire to enter data that are stored in the CAFE RDF triple store. During the data entry process, respondents receive feedback on how their institution compares to other institutions of the same type. These data are retrieved from the CAFE RDF triple store. All CAFE RDF data will be made available to implementation science and public health researchers. CAFE, Comparative Assessment Framework for Environments of Trauma Care; OOSTT, Ontology of Organizational Structures of Trauma centers and Trauma systems; RDF, Resource Description Framework.

    We addressed the second functionality by creating prototype visualizations which were evaluated during a focus group of trauma surgeons and trauma researchers at the 2018 ACS Clinical Congress in Boston. The visualizations have been refined based on the input from the focus group. The focus group also provided us with a list of research questions that potential users may wish to answer employing the data.

    In conceiving the CAFE project, one requirement was the ability to create flexible categories based on multiple organizational attributes and to program the CAFE system to automatically sort individual institutions or components into those categories. Utecht et al have demonstrated that Semantic Web Technologies (SWT) are a successful strategy for automatic data classification in sorting evidence types.19 Based on these considerations, we decided to build the CAFE data management system using SWT.

    SWT are methodologies that facilitate linked data and support running automated inferences on data. A core of SWT is the Resource Description Framework (RDF), a knowledge representation language. RDF consists of subject-predicate-object three-part statements (triples). RDF allows assigning a unique identifier to each entity in the domain of discourse. This permits one entity to be either the subject or the object in multiple triples. In effect, RDF enables the building of complex graphs out of the simple subject, predicate, and object structure. These graphs composed of triples are retained in RDF triple stores. The CAFE triple store is the central knowledge base component for this project.20

    To enable the full potential of automatic inference, the RDF triple store must contain background information about the organizational components and attributes relevant to the domain of trauma care and management. In an SWT environment this type of information is encoded in ontologies that are loaded into the RDF triple store along with the particular data to be analyzed. An ontology is a representation that is based on the real-world entities, processes, and relations of a given domain, rather than data. It represents the types of things in a given domain and how they are related to each other. For the domain of trauma centers, trauma systems, their organizational structures and processes, we created the Ontology of Organizational Structures of Trauma centers and Trauma systems (OOSTT) specifically for the CAFE project.21

    For the stage of the CAFE project discussed in this article, we used the SWT components in the following ways, which will be described in more detail below:

    • Retrieving definitions from OOSTT for display in the questionnaire.

    • Using semantic schemata to transform all answers into RDF triples.

    • Loading the RDF triples into the CAFE RDF triple store.

    • Extracting data from the CAFE RDF triple store to create figures for real-time comparative assessment.

    The key data collection tool of the CAFE project is the CAFE questionnaire (http://cafe-trauma.com/). Its purpose is to gather data about organizational structures of trauma centers and trauma systems while providing real-time comparison to those who enter the data. Data entry occurs through a response to specific questions about the trauma center’s structures and processes. Rolling the cursor over a bold-faced term brings up the definition used to guide respondents in consistent interpretation. The definitions are automatically retrieved from OOSTT’s underlying controlled vocabulary that was specifically created for CAFE.22 The terms for the controlled vocabulary were largely derived from the ACS Pre-Review Questionnaires (PRQ) used for trauma center verification and for trauma system consultations. Definitions were developed by a group of trauma system experts and then sent out for review to a group of trauma surgeons and subject matter experts. These carefully developed definitions enable a comparison of the user’s trauma center or system to all other trauma centers or systems that have entered the data using the same definitions to identify structures and processes.

    The trauma center questionnaire includes the following sections: basic trauma program information, trauma center personnel and their responsibilities, and specialty services (figure 2). Types of question responses include yes/no, multiple selection, and a drop-down box with response choices. As data are entered, the database builds the description of the trauma center in an RDF computer-interpretable language that will enable subtle similarities and differences between trauma centers to be distinguished. RDF statements created by the questionnaire tool are stored real time in the CAFE RDF triple store.23 To facilitate the progress being saved and general logistics of the questionnaire process, the answers are also recorded in a PostgreSQL database.24 Answering a question in the CAFE questionnaire creates one or more triples in the CAFE RDF triple store. In figure 3, ‘_:x’ is the object of one triple and the subject of five potential triples, only one of which is actually created based on the respondent’s answer.

    Figure 2
    Figure 2

    The CAFE questionnaire. The CAFE questionnaire is composed of Boolean (yes/no), multiple-choice (drop-down menu), or multiple selection (click boxes) questions. To facilitate orientation, the questionnaire is divided into three chapters with a total of 13 subsections. The right side of the screen shows the comparison of the answers given to other institutions of the same type that are already in the RDF triple store. CAFE, Comparative Assessment Framework for Environments of Trauma Care; RDF, Resource Description Framework; N/A, not applicable; TMD, trauma medical director.

    Figure 3
    Figure 3

    RDF statements created by answering the question ‘Who does the trauma medical director (TMD) report to?’ Answer choices are: ‘Hospital Board’, ‘Chief Executive Officer’, ‘Chief Medical Officer’, and ‘Head of Clinical Services’. This is a multiselection item, so the user may select multiple options. RDF, Resource Description Framework.

    The resulting information is pulled from the CAFE RDF triple store and used to build the comparison with other trauma centers in real time. The trauma center of interest will be compared with response averages from all trauma center respondents (figure 2). Data are deidentified when entered, so it is not possible to compare a trauma center with another specific trauma center.

    The CAFE RDF triple store was built using the RDF4J triple store.25 Utecht and Brochhausen have measured the usability of multiple RDF triple stores based on the CAFE requirements and used the results to make their selection.26

    Based on our goal of motivating potential users to share information about their institution, it is clear that the system needed to begin with an initial data set that could not be acquired from users. The American College of Surgeons Committee on Trauma (ACS-COT) facilitated this process. The ACS-COT received consent from three trauma centers and four trauma systems to use data from their trauma center verification reports or trauma system consultation reports to complete the questionnaire and to store and share the data in a deidentified manner in the CAFE system. Seven ACS staff members used the live questionnaire to enter data from the reports, providing us with feedback on the questionnaire and its functionalities.

    Results

    The data entry effort conducted by the ACS was successful and generated the CAFE RDF triple store holding a total of 2194 triples, which describe the organizational aspects of three trauma centers and four trauma systems.

    To confirm that the data management system is operational, we ran a set of test queries to ensure that the queries retrieved the correct answers. Queries were run to retrieve the answers to the following questions:

    • To whom does the trauma medical director (TMD) report?

    • At which level are the trauma centers verified?

    • How many certified registered nurse anesthetists (CRNA) are employed by each of the hospitals?

    • How many CRNAs are employed by each hospital that has an affiliated medical school?

    The results affirmed that we are able to correctly retrieve the information entered into the system by the ACS staff members.

    ACS staff provided feedback on the organization of the questions in the questionnaire and the definitions, which are pulled from OOSTT and provided as pop-ups during the questionnaire process. We revised the organization of the questionnaire and added more categories of questions to make the individual categories smaller and more consistent. To improve the definitions guiding the correct interpretation of trauma center structures and process, we initiated a thorough review of the definitions in OOSTT, relying on domain expert feedback. In total, our 142 terms and definitions received 1197 reviews. That response rate allowed us to have, on average, eight different people review each term and definition. Of the 1197 individual reviews, 986 considered the term and definition suitable, meaning that 82.37% of our definitions were deemed suitable. Definitions that were not deemed suitable have been edited.

    Next steps and discussion

    Table 1 gives an overview of the next steps for the CAFE project and projected tasks for a renewal that was recently submitted.

    View this table:
    Table 1

    Next steps

    One concern is who within a trauma center or trauma system would be responsible for entering data into the CAFE questionnaire. At the most basic level, answering the questions requires access to a large amount of data that must be obtained from disparate sources. At a more subtle level, the experience, perception, and level of engagement of the person answering the questions may affect the answers, as many require some nuance of interpretation. We assume that most of the data are readily available at trauma centers that have undergone ACS verification or trauma systems that have received ACS consultations. This is because the CAFE questionnaire and its terms were derived largely from information requested in the ACS trauma center verification process PRQ. In practice, the completion of the PRQ is often the responsibility of the trauma program manager (TPM) and the TMD, but is frequently a collaborative process involving many stakeholders from various areas in the hospital. In analogous fashion, the TPM and TMD may take responsibility for entering data about their trauma center’s structure and process into the CAFE questionnaire, but the responses may be more truly reflective if the input of a broader group is sought. The same process may be used by the TPM of a regional or state trauma system in collaboration with the system TMD.

    Since the CAFE questionnaire for trauma centers is based on the PRQ it would make sense to incorporate the CAFE questionnaire as part of the PRQ process rather than creating a redundancy. At over 300 questions, the PRQ is more extensive than the current version of the CAFE trauma center questionnaire (102 questions). Currently, the CAFE questionnaire includes only questions that are relevant to organizational structure, that do not identify the organization completing the questionnaire (eg, employee names, zip codes), and that do not have extensive text fields as answers. The major advantage of the CAFE questionnaire is that it collects data in a computer-parsable way by using a knowledge representation language. This allows the computer to draw inferences without human intervention and to manipulate the data immediately. The decision to implement an SWT-driven component of the PRQ process lies with the organizations and agency responsible for trauma center assessment and verification.

    At a future time, the CAFE project hopes to continue its collaboration with the ACS to link this database to outcome data, such as the TQIP database or another risk-adjusted benchmarking database. Once such a linkage occurs, the CAFE project will help fulfill the promise of investigating which trauma center or trauma system organizational structures and processes are most essential for certain patient outcomes.7 27

    Conclusion

    Based on the reported effort, we conclude that the CAFE questionnaire is ready to be used as a data collection tool to acquire organizational information about trauma centers and trauma systems. Visualization of comparative data supports a real-time assessment and comparison of the organizational structures of the participating trauma center or trauma system with organizations of the same kind that are already in the database. The collected data are well curated and maintained, and data can be obtained from the CAFE RDF triple store by running queries.

    Acknowledgments

    The authors thank the anonymous reviewers for their insightful comments.

    References

    1. 1.↵
      1. American College of Surgeons, Committee on Trauma
      . Resources for optimal care of the injured patient. Chicago, IL: American College of Surgeons, Committee on Trauma, 2014.
    2. 2.↵
      1. Hemmila MR,
      2. Cain-Nielsen AH,
      3. Jakubus JL,
      4. Mikhail JN,
      5. Dimick JB
      . Association of hospital participation in a regional trauma quality improvement collaborative with patient outcomes. JAMA Surg 2018;153:747–56.doi:10.1001/jamasurg.2018.0985pmid:http://www.ncbi.nlm.nih.gov/pubmed/29800946
      OpenUrlPubMed
    3. 3.↵
      1. Boyd JM,
      2. Moore L,
      3. Atenafu EG,
      4. Hamid JS,
      5. Nathens A,
      6. Stelfox HT
      . A retrospective cohort study of the relationship between quality indicator measurement and patient outcomes in adult trauma centers in the United States. Injury 2017;48:13–19.doi:10.1016/j.injury.2016.10.040pmid:http://www.ncbi.nlm.nih.gov/pubmed/27847191
      OpenUrlPubMed
    4. 4.↵
      1. Heaney JB,
      2. Guidry C,
      3. Simms E,
      4. Turney J,
      5. Meade P,
      6. Hunt JP,
      7. McSwain NE,
      8. Duchesne JC
      . To TQIP or not to TQIP? that is the question. Am Surg 2014;80:386–90.doi:10.1177/000313481408000422pmid:http://www.ncbi.nlm.nih.gov/pubmed/24887671
      OpenUrlPubMed
    5. 5.↵
      1. Hemmila MR,
      2. Jakubus JL
      . Trauma quality improvement. Crit Care Clin 2017;33:193–212.doi:10.1016/j.ccc.2016.08.010
      OpenUrl
    6. 6.↵
      1. Winchell RJ,
      2. Eastridge BJ,
      3. Moore MM,
      4. Ashley DW,
      5. Gaines BA,
      6. Gainor D,
      7. Jahangir AA,
      8. Krieg JC,
      9. Mays CA,
      10. Michaels HN, et al
      . Developing a national trauma system: proposed governance and essential elements. J Trauma Acute Care Surg 2018;85:637–41.doi:10.1097/TA.0000000000001994pmid:http://www.ncbi.nlm.nih.gov/pubmed/29851911
      OpenUrlPubMed
    7. 7.↵
      1. Moore L,
      2. Evans D,
      3. Hameed SM,
      4. Yanchar NL,
      5. Stelfox HT,
      6. Simons R,
      7. Kortbeek J,
      8. Bourgeois G,
      9. Clément J,
      10. Lauzier F, et al
      . Mortality in Canadian trauma systems: a multicenter cohort study. Ann Surg 2017;265:212–7.doi:10.1097/SLA.0000000000001614pmid:http://www.ncbi.nlm.nih.gov/pubmed/28009748
      OpenUrlPubMed
    8. 8.↵
      1. Trooskin SZ,
      2. Copes WS,
      3. Bain LW,
      4. Peitzman AB,
      5. Cooney RN,
      6. Jubelirer RA
      . Variability in trauma center outcomes for patients with moderate intracranial injury. J Trauma 2004;57:998–1005.doi:10.1097/01.TA.0000135349.24414.AEpmid:http://www.ncbi.nlm.nih.gov/pubmed/15580023
      OpenUrlPubMed
    9. 9.↵
      1. Shafi S,
      2. Friese R,
      3. Gentilello LM
      . Moving beyond personnel and process: a case for incorporating outcome measures in the trauma center designation process. Arch Surg 2008;143:115–9.
      OpenUrlCrossRefPubMedWeb of Science
    10. 10.↵
      1. Wade CE,
      2. del Junco DJ,
      3. Holcomb JB,
      4. Holcomb JB,
      5. Wade CE,
      6. Brasel KJ,
      7. Vercruysse G,
      8. MacLeod J,
      9. Dutton RP, et al. Trauma Outcomes Group
      . Variations between level I trauma centers in 24-hour mortality in severely injured patients requiring a massive transfusion. J Trauma 2011;71:S389–93.doi:10.1097/TA.0b013e318227f307pmid:http://www.ncbi.nlm.nih.gov/pubmed/21814110
      OpenUrlPubMed
    11. 11.↵
      1. Shafi S,
      2. Stewart RM,
      3. Nathens AB,
      4. Friese RS,
      5. Frankel H,
      6. Gentilello LM
      . Significant variations in mortality occur at similarly designated trauma centers. Arch Surg 2009;144:64–8.doi:10.1001/archsurg.2008.509pmid:http://www.ncbi.nlm.nih.gov/pubmed/19153327
      OpenUrlCrossRefPubMedWeb of Science
    12. 12.↵
      1. Cudnik MT,
      2. Sayre MR,
      3. Hiestand B,
      4. Steinberg SM
      . Are all trauma centers created equally? A statewide analysis. Acad Emerg Med 2010;17:701–8.doi:10.1111/j.1553-2712.2010.00786.xpmid:http://www.ncbi.nlm.nih.gov/pubmed/20653583
      OpenUrlCrossRefPubMedWeb of Science
    13. 13.↵
      1. Minei JP,
      2. Schmicker RH,
      3. Kerby JD,
      4. Stiell IG,
      5. Schreiber MA,
      6. Bulger E,
      7. Tisherman S,
      8. Hoyt DB,
      9. Nichol G, et al. Resuscitation Outcome Consortium Investigators
      . Severe traumatic injury: regional variation in incidence and outcome. Ann Surg 2010;252:149–57.doi:10.1097/SLA.0b013e3181df0401pmid:http://www.ncbi.nlm.nih.gov/pubmed/20531005
      OpenUrlCrossRefPubMedWeb of Science
    14. 14.↵
      1. Lerner EB,
      2. Willenbring BD,
      3. Pirrallo RG,
      4. Brasel KJ,
      5. Cady CE,
      6. Colella MR,
      7. Cooper A,
      8. Cushman JT,
      9. Gourlay DM,
      10. Jurkovich GJ, et al
      . A consensus-based criterion standard for trauma center need. J Trauma Acute Care Surg 2014;76:1157–63.doi:10.1097/TA.0000000000000189
      OpenUrlCrossRefPubMed
    15. 15.↵
      1. Uribe-Leitz T,
      2. Esquivel MM,
      3. Knowlton LM,
      4. Ciesla D,
      5. Lin F,
      6. Hsia RY,
      7. Spain DA,
      8. Winchell RJ,
      9. Staudenmayer KL
      . The American College of surgeons needs-based assessment of trauma systems: estimates for the state of California. J Trauma Acute Care Surg 2017;82:861–6.doi:10.1097/TA.0000000000001408pmid:http://www.ncbi.nlm.nih.gov/pubmed/28248801
      OpenUrlPubMed
    16. 16.↵
      1. Jansen JO,
      2. Moore EE,
      3. Wang H,
      4. Morrison JJ,
      5. Hutchison JD,
      6. Campbell MK,
      7. Sauaia A
      . Maximizing geographical efficiency: an analysis of the configuration of Colorado's trauma system. J Trauma Acute Care Surg 2018;84:762–70.doi:10.1097/TA.0000000000001802pmid:http://www.ncbi.nlm.nih.gov/pubmed/29370062
      OpenUrlPubMed
    17. 17.↵
      1. Moore L,
      2. Champion H,
      3. Tardif P-A,
      4. Kuimi B-L,
      5. O’Reilly G,
      6. Leppaniemi A,
      7. Cameron P,
      8. Palmer CS,
      9. Abu-Zidan FM,
      10. Gabbe B, et al
      . Impact of trauma system structure on injury outcomes: a systematic review and meta-analysis. World J Surg 2018;42:1327–39.doi:10.1007/s00268-017-4292-0
      OpenUrl
    18. 18.↵
      1. Leenstra NF,
      2. Jung OC,
      3. Johnson A,
      4. Wendt KW,
      5. Tulleken JE
      . Taxonomy of trauma leadership skills: a framework for leadership training and assessment. Acad Med 2016;91:272–81.doi:10.1097/ACM.0000000000000890pmid:http://www.ncbi.nlm.nih.gov/pubmed/26352763
      OpenUrlPubMed
    19. 19.↵
      1. Utecht J,
      2. Brochhausen M,
      3. Judkins J,
      4. Schneider J,
      5. Boyce RD
      . Formalizing evidence type definitions for drug-drug interaction studies to improve evidence base curation. Stud Health Technol Inform 2017;245:960–4.pmid:http://www.ncbi.nlm.nih.gov/pubmed/29295242
      OpenUrlPubMed
    20. 20.↵
      1. Antoniou G,
      2. Groth P,
      3. van HF
      . A semantic web primer. Cambridge, MA: MIT Press, 2012.
    21. 21.↵
      1. Utecht J,
      2. Judkins J,
      3. Otte JN,
      4. Colvin T,
      5. Rogers N,
      6. Rose R,
      7. Alvi M,
      8. Hicks A,
      9. Ball J,
      10. Bowman SM, et al
      . OOSTT: a resource for analyzing the organizational structures of trauma centers and trauma systems. CEUR Workshop Proc 2016;1747:1747.pmid:http://www.ncbi.nlm.nih.gov/pubmed/28217041
      OpenUrlPubMed
    22. 22.↵
      1. Utecht J,
      2. Ball J,
      3. Bowman SM,
      4. Dodd J,
      5. Judkins J,
      6. Maxson RT,
      7. Nabaweesi R,
      8. Pradhan R,
      9. Sanddal ND,
      10. Winchell RJ, et al
      . Development and validation of a controlled vocabulary: an owl representation of organizational structures of trauma centers and trauma systems. Stud Health Technol Inform 2019;264:403–7.doi:10.3233/SHTI190252pmid:http://www.ncbi.nlm.nih.gov/pubmed/31437954
      OpenUrlPubMed
    23. 23.↵
      1. W3C
      . RDF 1.1 Primer. 2014. https://www.w3.org/TR/2014/NOTE-rdf11-primer-20140624/ (Accessed 28 Jan 2020).
    24. 24.↵
      1. The PostgreSQL Global Development Group
      . PostgreSQL. https://www.postgresql.org/ (Accessed 28 Jan 2020).
    25. 25.↵
      1. Eclipse Foundation
      . Eclipse RDF4J. https://rdf4j.eclipse.org/ (Accessed 28 Jan 2020).
    26. 26.↵
      1. Utecht J,
      2. Brochhausen M
      . Measuring the usability of triple stores for knowledge management on trauma care organizations. CEUR Workshop Proc 2015;1546:241–2.
      OpenUrl
    27. 27.↵
      1. Moore L,
      2. Champion H,
      3. O’Reilly G,
      4. Leppaniemi A,
      5. Cameron P,
      6. Palmer C,
      7. Abu-Zidan FM,
      8. Gabbe B,
      9. Gaarder C,
      10. Yanchar N, et al
      . Impact of trauma system structure on injury outcomes: a systematic review protocol. Syst Rev 2017;6:12. doi:10.1186/s13643-017-0408-8

    Footnotes

    • Contributors As PI of the CAFE project MB created the outline of the CAFE system and planned and supervised its implementation and assessment. He is the curator of the Ontology of Organizational Structures of Trauma centers and Trauma systems (OOSTT) and the main author of the current article. JB and NDS were key contributors to the development of the CAFE system and its implementation and contributed to the research and writing of the background section of this article. JD led the CAFE-related research at the American College of Surgeons Committee on Trauma. He supervised the initial data collection for the CAFE system at the ACS. NB provided technical support in transferring the CAFE code from the University of Arkansas for Medical Sciences to the University of Florida. In this capacity she made modifications to the CAFE code and environment. SB is responsible for CAFE reporting and contributed by analyzing and presenting the research process during the years. She also provided copyediting for the current publication. JU is the key software developer who developed the CAFE system in collaboration with MB. He also conducted the assessment of the OOSTT definitions by subject matter experts. RJW provided input from his experience as a trauma surgeon and ACS-COT official during all aspects of planning, implementation, and evaluation of the CAFE system. KWS provided insight on aspects of implementation, evaluation and future use of the ACS system. All authors contributed to the writing of the article and reviewed its content.

    • Funding The research presented in this article is funded by the National Institute of General Medical Sciences of the National Institutes of Health under award number R01GM111324.

    • Competing interests None declared.

    • Patient consent for publication Not required.

    • Ethics approval This research was deemed exempt by the University of Florida Institutional Review Board per IRB201902564.

    • Provenance and peer review Not commissioned; externally peer reviewed.