Validation of a predictive model for operative trauma experience to facilitate selection of trauma sustainment military–civilian partnerships

Background

Trauma case volumes can be random and unpredictable, even at high-volume centers, making planning and implementation of trauma training challenging. As the Department of Defense (DoD) fully engages in trauma sustainment military–civilian partnerships (TS-MCP) to address readiness gaps, the ability to accurately plan for training schedules and set realistic expectations for trauma experience is needed. While methods using complicated artificial neural networks have shown limited success, a simpler model is required for practical application.1 This group has previously described straightforward prediction tool to optimize planning at TS-MCP to maximize the operative trauma experience but this model had not prospectively tested.2 The objective of this study is to evaluate the accuracy of this method in a prospective manner, by applying the method’s predictions to planned trauma immersion courses. By demonstrating its prospective accuracy, we aim to demonstrate a predictive model’s usefulness for planning trauma operative exposure throughout the year.

Methods

A method estimating the number of emergent trauma cases by specialty was designed using 2 years of retrospective trauma data to predict case exposure for the following year.2 The method as described allows for estimating with a 95% level of confidence the estimated number of cases a surgeon would experience after any number of 24-hour shifts or the number of shifts that would be required to reach a minimum number of cases. This model was applied to four ACS level 1 trauma centers: Saint Louis University Hospital, Barnes-Jewish Hospital, San Antonio Military Medical Center (SAMMC), and R Adams Cowley Shock Trauma Center. These centers were selected based on their current participation as military trauma training sites and willingness to provide trauma data based on the requirements of the predictive model. This made each institution similar in engagement potential for rotating surgeons, but not necessarily similar in case volume. At each center, the 2 years of retrospective data were analyzed, and prospective estimates were made for the third year using the model.

To test the model and demonstrate its application in trauma course planning, a theoretic schedule of trauma immersion courses was designed and scheduled for the third (prospective) year. Courses were designed to be 2–4 weeks in duration. The model used implemented five 24-hour call periods over the 2 weeks to estimate a potential, although rigorous, call schedule (an estimated Q3-4 call schedule). Similarly, a 4-week course would include 12 call periods. Call periods would be staggered in a similar fashion for the 4-week course as done in the 2-week course. The maximum number of call periods that could fit into a 2-week and 4-week block were not used for this analysis given the real-world administrative burdens of these courses. For example, if rotators arrived on a Monday, they would likely require administrative and educational sessions prior to taking call, this lag-time was taken into account in the analysis. The actual cases seen on the five 24-hour shifts and twelve 24-hour shifts for the 2-week and 4-week courses were compiled respectively and compared with the model’s predictions for accuracy. In all, 51 separate 2-week courses and 49 separate 4-week courses were created over the academic year.

The designed courses were notional for the purposes of evaluating the model; however, the number of operative cases that would have occurred during call periods were measured based on the previously defined method of including a case. An operative trauma case for any given calendar day would count if the patient arrived and was operated on within one calendar day of arriving. For example, a patient arriving on October 4, 2017 would count as an urgent operative case on October 4 if he had an operation on October 4 or October 5, 2017. This method was chosen to best estimate “urgent cases,” those requiring operative intervention either emergently or within a day. It allowed capture of cases done over midnight and early morning, whereas using a 24-hour timeline led to inaccuracies in registry data retrieval (data are captured by calendar day). Additionally, having rotating staff involvement introduces the challenge of decreasing the experience for the organic staff and Graduate Medical Education (GME) participants; therefore, if a robust enough case volume cannot be sustained for the rotators and organic staff, the estimated length of a course would expected to be longer.

During the development of the model, seasonal variations were observed at each institution except SAMMC. For this analysis, the seasonal ranges were collapsed from four seasons to a more practical two, with the summer range including March–November, and the winter range including December–February. Using this method, 13 winter courses and 38 summer courses were created for the 2-week courses. Splitting the year into 4-week courses yielded 12 winter courses and 37 summer courses.

The predictive model was applied for a full year at R Adams Cowley Shock Trauma Center, Barnes-Jewish Hospital, and Saint Louis University Hospital. The notional courses were designed for the year at these institutions starting October 1, 2017 to September 30, 2018. SAMMC had only 8 months of data to test at the time of this study; therefore, a proportionally reduced number of courses included in this analysis. Poisson regression was applied to the three sites with a full year of data to assess any differences in count of cases depending on the hospital.

Results

During a 12-month period, the total number of urgent cases that occurred at R Adams Cowley Shock Trauma Center was 913 orthopedic cases and 465 trauma/general surgery cases. At Barnes-Jewish hospital, there were 818 urgent orthopedic cases and 263 urgent trauma/general surgery cases. At Saint Louis University Hospital, there were 427 urgent orthopedic cases and 188 urgent trauma/general surgery cases. For both trauma and orthopedics, R Adams Cowley Shock Trauma Center has significantly higher case volume than the other two sites (p<0.0001) during a 12-month period. The predicted volume of urgent cases for trauma/general surgery fell within 96%–98% of the predicted ranges for all the 2-week and 4-week courses at all institutions (table 1). Orthopedic operative exposure was predicted less accurately, with the lowest accuracy of 82%. The distribution of cases for the 4-week and 2-week courses are demonstrated in figures 1 and 2

Request permission

Request permission

During an 8-month period, the total number of cases that occurred at SAMMC was 336 orthopedic and 165 trauma/general surgery cases. The model had 34 2-week and 32 4-week theoretical courses at SAMMC January 1, 2018 to August 31, 2018. There was not a predicted seasonal difference at SAMMC. For 2-week courses, the predicted case volume for an individual course was 0–9 for trauma/general surgery and 2–12 for orthopedics. For the 4-week courses, the case volume ranges were 3–17 for trauma/general surgery and 8–24 for orthopedics. Course case volume would have fallen into the predicted range 94% of the time for orthopedics and 98% of the time in trauma/general surgery for the 2-week course. For the 4-week course, the accuracy was 87% and 98% for trauma/general surgery and orthopedics, respectively.

Discussion

On June 4, 2019, the US House of Representatives passed The Pandemic and All-Hazards Preparedness and Advancing Innovation (PAHPAI) Act of 2019; the Mission Zero Act is included in the PAHPAI. This legislation includes the implementation of the recommendations from the June 2016 National Academy of Sciences, Engineering and Medicine report, A National Trauma Care System: Integrating Military and Civilian Trauma Systems to Achieve Zero Preventable Deaths After Injury. The Mission Zero Act builds on the legislative framework from the 2017 National Defense Authorization Act by supporting the incorporation of military trauma care providers into the civilian setting. TS-MCPs are now supported by law to increase military trauma care readiness.3 As the DoD enters into partnerships with civilian trauma centers having practical prediction models to assess the “readiness value” of TS-MCP is imperative and objective measurements are a starting point for this determination.4 5 Predictable educational experiences facilitate planning, expectation management and help inform/establish the goals of the trauma skills sustainment training. Having a method to potentially predict a case-volume threshold in terms of location, timing length of courses will be informative for site selection as well as planning purposes. These results support the use of our previously described tool to predict the operative trauma volume at TS-MCP courses.

While operative experience is not the sole factor in determining the most appropriate TS-MCP, case volume cannot be underestimated as foundational to preparing a team or provider for surgical care of the combat casualty. Many military treatment facilities report low operative and trauma volumes, resulting in military providers that lack exposure to the critical skills needed to care for severely injured combat casualties.6–8 In addition, a model that accurately predicts trauma case volume at an institution would be valuable in establishing TS-MCP courses. This method, as demonstrated in this study, provides an accurate range of possibilities to allow for planning. Most importantly, it provides an accurate minimum number of cases. While only providing a rough prediction of the number of cases a rotator will experience, through this method a minimum number of operative cases can be predicted with some certainty. If a “trauma volume threshold” becomes a deployment standard, this model would facilitate planning at a readiness site. Additionally, this method also provides some seasonal information on trauma volumes that can further allow for educational planning to include potentially increasing the duration of courses in winter months.

An ideal scenario would be to test a similar predictive model for individual types of cases. As the military adopts a knowledge Skills Abilities metric for readiness, which places a “point” value on specific types of cases, this model may help predict the timeline to skills or abilities readiness. Unfortunately, this prediction is not very practical for specific types of cases. As an example, earlier research at Saint Louis University identified 76 urgent operative vascular cases over the course of 731 days, an overall rate of 0.1 vascular cases per day.9 Other institutions may have much higher volume of urgent operative vascular cases that make predictive modeling for a course practical, but this is unlikely. Specific experience with complex vascular trauma cases, and other cases that improve combat casualty care readiness will likely have to be obtained through training courses such as ACS ASSET and Emergency War Surgery course which is being modified to support a military trauma needs.

The most significant weakness in this model is the lack of data on specific times of the day or night for operative cases. Using a calendar day, patients could in theory arriving shortly after midnight on 1 day and be operated on shortly before midnight the following day and count for the first day. Though this possible scenario is an uncommon event, it remains an uncertainty. This inherent inaccuracy in the calculation could account for the decreased accuracy of the model with orthopedic cases, as orthopedic injuries are seldom life threatening and interventions are more likely to be delayed to the following day or beyond for definitive treatment. This illustrates that the practical application or predictive methodology can be challenging, and consistent data measurement is imperative. The data in this analysis were from trauma registries which can be recorded using different software and accessed by different personnel at different times. Any variation in data collection or retrieval methods can alter predictions and outcomes. Therefore, there would be further improvement in this model if higher fidelity timelines were possible, including accurate times for patient arrival and OR start times to better predict the number of surgical cases during a rotation. Additional analysis of factors that affect orthopedic operative case timing would also be prudent to improve orthopedic accuracy.

There are a multitude of factors that contribute to the success of a TS-MCP. While this model provides a very quantitative assessment, there are significant qualitative factors as well. Other crucial attributes to a successful TS-MCP include leadership commitment, departmental support, administrative maintenance, competing GME interests, geographic location, exposure to different damage control strategies (resuscitation and surgical), length of courses, and having permanent military cadre present. While the aforementioned attributes of a successful TS-MCP are crucial, ultimately trauma readiness for military surgeons relies on an appropriate level of experience and operative volume. Additionally, the acuity of the cases has to be taken into consideration. High acuity operative cases managing multi-cavity hemorrhage in patients that require massive transfusion are extremely valuable for military trauma readiness. This model does not address the acuity of the case and used the time course of surgical intervention as a surrogate for urgency. If this model proves to be useful when used by the DoD, additional variates such as blood transfusion, injury severity score, and procedure codes can be incorporated into the model to provide not just case volume but military-relevance case acuity. If more specific requirements are added though, invariably the predicted length of time required in any course will go up if very specific operative experiences are required. For visiting surgeons that are expecting to deploy, the length of courses impacts their practices and families at home.

This study prospectively validates our previously described method for predicting trauma case volume at TS-MCP. Two previous publications used different methodologies that arrived at similar predictions.2 9 The methods demonstrated in this paper are pragmatic, straightforward, and easily applicable. No model can predict the extremes however, and trauma centers can have seemly random lulls in case volumes; therefore, TS-MCPs and the mission requirements that govern them must have built in contingency plans like simulation, cadaver training, extended courses, or didactics. Model development and refinement should be ongoing as more data are gathered and a learning model developed that changes as case volume and case acuity change. Further refinement and modification will hopefully be able to predict trauma and guide course structure to eliminate situations where rotators fail to meet their operative case goal. Prospective application of the model in its current state for validation of existing TS-MCP and guidance for determination of future TS-MCP is expected to validate and lead to further refinement.

This model can potentially be used as a predictive tool for the assessment of surgical trauma opportunities at current or future TS-MCPs. Unfortunately, experienced case volume is not a direct surrogate for competency and quality of care of a surgeon. Additional assessment and validation tools will need to be developed to augment this model and help determine competency of surgeons. Robust trauma performance improvement program already exist at many sites that would be considered for TS-MCPs; therefore, rotators should be required to participate in these program. Objective competency assessments, especially during short rotations, are challenging and should be developed, studied, and refined in conjunction with ongoing DoD efforts. Research efforts are also necessary to determine the ideal length of time for these program and how to best assess and train surgeons for wartime skills, despite inherent variance in competencies. With this model, possible TS-MCPs sites that have a predictably large number of urgent surgical cases can be identified. If there are predictable surgical trauma cases, the opportunity to evaluate rotators for competency while they operate on those cases also exists.

Conclusion

The model’s utility for predicting operative trauma for the purpose of designing courses at TS-MCP is high for trauma/general surgery, but the accuracy diminishes substantially for orthopedic trauma. Predictive models for operative trauma experience can inform the selection of TS-MCP.