Talks I Have Given••

Advanced and alternative research methods for the acute care surgeon scientist

Abstract

Clinical research has evolved significantly over the last few decades to include many advanced and alternative study designs to answer unique questions. Recognizing a potential knowledge gap, the AAST Associate Member Council and Educational Development Committee created a research course at the 2022 Annual Meeting in Chicago to introduce junior researchers to these methodologies. This manuscript presents a summary of this AAST Annual Meeting session, and reviews topics including hierarchical modeling, geospatial analysis, patient-centered outcomes research, mixed methods designs, and negotiating complex issues in multicenter trials.

Introduction

Historically, most clinical research studies in the medical literature fell into one of only a few distinct categories, including ecological, case-control, cohort, and randomized controlled trials. Recently, the realization that more statistically robust methodologies can provide richer analyses and interpretation has contributed to the increased utilization of these advanced techniques. Acute care surgeon scientists should be familiar with these advanced techniques, which are being more frequently used for research in the trauma and acute care surgery realm. The Associate Member Council and the Educational Development Committee of the American Association for the Surgery of Trauma developed an in-person session for the 2022 Annual Meeting in Chicago, Illinois to discuss these topics. This article includes the proceedings from that scientific session.

Cutting edge health services research techniques

Hierarchical modeling

Hierarchical models are perhaps the most pervasive statistical tools to have gained common usage in trauma research over the past decade. However, many researchers may not recognize their uses. Regression models (eg, linear, logistic, Poisson) rely on the assumption that all observations in a dataset are independent: characteristics and outcomes for each observation are not related. This condition is often violated in health services research due to clustering. The most common example in trauma research is the clustering of patients within trauma centers (TCs) (eg, in the National Trauma Data Bank), but there are many other important examples (eg, vehicle occupants within crashes, TCs within cities, counties within states). It is incumbent on the researcher to determine if clustering is present in their own study. If clustering is present, it is statistical best practice to account for it in a hierarchical, or multilevel, model.1 2

Beyond a statistical necessity, hierarchical models have many practical uses to research. First, they can be used to compare variations in practice or outcomes between TCs. This is done by selecting the outcome of interest, for example, “time-to-surgery” or “mortality,” and using the hierarchical model to output the ‘cluster-level’ effects: the average change in time-to-surgery or odds of mortality for each TC. This yields a caterpillar plot and is the same methodology used by the Trauma Quality Improvement Program to provide risk-adjusted TC performance measures.3

Second, ‘measures of variance’ are lesser known yet powerful tools to quantify variation. These include the median OR, proportional change in variance, and intraclass correlation coefficient. Table 1 defines these measures, their practical uses, and interpretation.

Table 1
|
Health services research techniques with applications to trauma and acute care surgery research

Advanced confounder control

Propensity score (PS) and instrumental variable (IV) methodologies are often termed ‘novel’ methods for confounder control (as opposed to traditional regression methods). In PS analyses, the PS represents the probability of receiving the treatment of interest (exposure) and is derived for each patient using a logistic regression model accounting for all conceivable factors that might contribute to selection bias. As a result, the PS is thought of as a ‘balancing score’ that can be used in matched or weighted analyses to measure the ‘true’ association between treatment and outcome. Unfortunately, PS analyses are often performed and interpreted poorly. For example, authors often claim that PS matching ‘eliminates confounding’—this is untrue. We recommend careful attention to the correct use, reporting, and interpretation of these methods.4

IVs are a powerful means of controlling for confounding in observational studies that are less frequently used. An IV is defined as a factor that is strongly correlated to the treatment under study, but not related to the outcome. The only way that an IV acts on the outcome is through the allocation of the treatment. A true IV is very difficult to find in available trauma datasets. For this reason, authors often create an ‘area-level’ measure to act as an IV—for example, the county % of direct-to-TC transport in a study evaluating impact of direct transport to TC on survival,5 or the hospital percentage of early amputation in a study evaluating impact of limb salvage strategies on mortality.6

Geospatial analysis

Trauma researchers should consider using geospatial analysis when location information is available and the objective is to evaluate the relationship between points on a map. The first decision in designing a study using geospatial analysis is the geographic unit of measurement. These range from the points on a map themselves (eg, locations of injury), to aggregating events within larger geographic areas such as census tracts, zip codes, counties, cities, or states. Evaluating access-to-care is a primary use of geospatial analysis in trauma research, and is enhanced through use of network analysis to estimate true distance or time to trauma care along roads for individuals7 or populations of people.8 Hot spot analysis is another essential tool for identifying clusters or regions that are statistical high or low outliers in trauma outcomes, with important implications for trauma system performance.9 Geospatial approaches often use physical distances, however, time-based approaches (ie, transport time to a TC) might also be useful.10

Patient-centered outcomes research

If we ask ourselves the question, “Why are we doing research and what are we hoping to accomplish?”, the answer will most likely relate to our desire to have an impact. One of the ways to have an impact is by using the lens of our patients and other stakeholders and focusing on the outcomes that they perceive to be most relevant. This principle is central to doing research that is patient centered.11 Patient-centered outcomes research focuses the attention on the patient’s beliefs, preferences, and needs, in contrast to physician-centered care. Thus, active participation of the patient as a stakeholder is an essential element of patient-centered outcomes research (PCOR).12 Additionally, PCOR engages stakeholders and focusing on outcomes and processes that are prioritized by individuals that are impacted the most which allows for upfront focus on effective implementation and dissemination.

When it comes to PCOR, the Patient-Centered Outcomes Research Institute (PCORI) dominates. In 2010, Congress authorized the establishment of PCORI. It is a publicly supported, independent, non-profit research institute which is devoted to funding comparative clinical effectiveness research (CER) that addresses questions and concerns important to patients. PCORI funds projects that generate and disseminate evidence that is ‘relevant, trustworthy, and useful’ to patients and others its serves. By 2021, PCORI had invested >US$3.4 billion to fund patient-centered, stakeholder-engaged CER studies and other initiatives. Despite concerns that PCORI may not be reauthorized, in 2019, Congress did reauthorize PCORI funding for another 10 years. PCORI’s strategic goals13 are to:

  • increase quantity, quality, and timeliness of useful, trustworthy research information available to support health decisions;

  • speed the implementation and use of patient-centered outcomes research evidence;

  • influence research funded by others to be more patient centered.

Through a series of ‘National Priorities for Health’ that ‘anchor the work’, the PCORI Research Agenda serves as a guide to the development of the research portfolio of PCORI-funded CER projects. PCORI is not the only funding agency interested in PCOR. For example, the Agency for Healthcare Research and Quality partners with PCORI and The Department of Health and Human Services through the Patient-Centered Outcomes Research Trust Fund to disseminate evidence about PCOR and CER.14

Patient and stakeholder engagement is key to the development of successful PCOR projects. Given the central role of the patient perspective in PCOR, methods to incorporate patient-reported outcomes (PROs) are critical. PROs are defined by the Food and Drug Administration and National Quality Form (NQF) as ‘… a report that comes directly from the patient … about the status of a patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else’.15 16 PROs are distinct from other patient outcomes such as physiological measures, process measures, clinician-reported measures, and caregiver-reported measures. A good PRO should include the following attributes: non-intrusive, realistic, relevant, reached by consensus, and easy to understand.17 Other methodology to incorporate patient-focused metrics and outcomes include strategies such as use of a discrete choice experiments (DCE) to help determine the best primary and secondary aims of a study.18 DCE are a quantitative technique used to measure individual preferences by administering surveys that then ask individuals to choose options between two or more hypothetical scenarios.19 This allows for patient preference to be directly linked to an outcome measure of interest.

In 2021, the Coalition for National Trauma Research received a PCORI Engagement Award to create The Community of Trauma Care—Partnering with Patients and Caregivers to Improve Injury Outcomes. The objectives of this partnership are to:

  • establish stakeholder Injury Research Engagement Panel (I-REP) to partner in PCOR/CER from conception to dissemination;

  • develop emergency-setting informed consent strategies and methods for increasing follow-up engagement;

  • conduct reciprocal education between patients and researchers.

The project seeks to create a sustainable I-REP to engage stakeholders in trauma research, ensuring appropriate research methods and questions, to develop patient/family-centered communications on participating in trauma PCOR/CER, and to disseminate findings to stakeholders to PCOR/CER to improve engagement strategies and outcomes.

Mixed methods

Mixed methods research methods purposefully and intentionally combine both quantitative and qualitative data to study a research question.20 In the field of trauma and acute care surgery, the vast majority of studies have been performed using quantitative research, or the study of numbers with the applications of statistics to determine the amplitude of effect sizes. While quantitative research can reach conclusions about questions such as ‘whether’ and ‘how much’, these analyses sometimes lack the ability to answer questions about ‘why’ or ‘how’. Qualitative research uses the collection of non-numerical data, such as text, video, or audio to understand concepts, opinions, and experiences. Common data sources for qualitative research include interviews, focus groups, and surveys, which can be used to gather in-depth insights or generate new ideas. Qualitative studies can help us understand why an intervention does not work in the real world, how patients experience care, and how practitioners think.21 22

There are a number of approaches to mixed methods study design, and some of the most commonly used include: triangular (also known as convergent), exploratory sequential, explanatory sequential, and embedded or nested (figure 1).23 These designs use integration, or mixing, of results of the quantitative and qualitative results at different points of the study and can be tailored to the specific research question.24

Figure 1
Request permissionrequest permission icon
Figure 1

This is a representation of the four common approaches to mixed methods research: (1) triangular design, (2) exploratory sequential design, (3) explanatory sequential design, and (4) embedded design. The exploratory and explanatory sequential designs are named for the qualitative component.

  1. Triangular: quantitative and qualitative data are collected simultaneously and the results are mixed and interpreted together. In general, the interpretation of results emphasizes the results from both methods.

  2. Exploratory sequential: qualitative data analysis occurs first, and the analysis of qualitative data is used to develop a theory or an instrument which can be used or tested quantitatively. Quantitative analysis can help to generalize or test qualitative results.

  3. Explanatory sequential: quantitative data analysis occurs first, and the results are used to develop a qualitative data collection strategy. Here, the quantitative results can be clarified, explained, or elaborated using the qualitative results.

  4. Embedded: a quantitative design is the main study methodology, and subjects from the study are enrolled in a nested qualitative study, where the qualitative analysis enhances or explains findings from the quantitative study. Here, the interpretation generally focuses on the quantitative outcomes.

A key justification for the use of mixed methods in research of health-related questions is to examine diverse types of research questions, such as how patients experience health conditions and care, how organizations provide care, as well as the context and complexity of decisions and outcomes.25 Mixed research methods can provide a more complete picture than a standalone quantitative or qualitative study, as their strengths and weaknesses complement one another. Advantages of mixing methods over using either method alone include contextualization, as the two methods can uncover data which is dissonant and help explain discrepancies, credibility, as the two methods can enhance the validity of findings if they reach similar findings, and generalizability, as quantitative methods can improve external validity of a qualitative study. Teams with diverse methodological training are instrumental to the design and implementation of mixed methods studies.

Negotiating complex issues in multicenter trials

Multicenter clinical trials (MCT) present an important collaborative opportunity to accelerate clinical discovery and enroll diverse patient population. Although the benefits of MCTs are substantial, complex dynamics and operational issues can challenge successful execution. Six major clinical trial content areas (table 2) were presented, and an example issue within that content area was reviewed.

Table 2
|
These are examples of six major content areas related to clinical trials and example issues that are discussed in the manuscript related to each content area.

Implementation of an MCT is heavily dependent on adequate funding to support trial operations. While numerous funding agencies exist, federally funded trials are a hallmark of academic research. Conventionally, MCTs are funded by grant mechanisms that provide funding in support of trial operations. More recently, many federal agencies, including the National Institutes of Health and United States Department of Defense have introduced a funding mechanism called other transactional authority (OTA). An ‘other transaction is a type of legal instrument with a series of unique regulations than may differ from contracts and grant. Although a full discussion of these differences is beyond the scope of this commentary, it is imperative for MCT investigators considering an OTA mechanism to understand the structure and regulations of this approach.26

Implementation of an MCT can be challenging due to the involvement of multiple investigators and institutions. Although these features are assets in the execution of a trial, they can present challenges for communication and organization. One potential option for overcoming this barrier is to use existing clinical trials networks. These networks, often combining efforts of multiple societies and institutions, carry precedent in coordinating multiple sites and investigators. Existing trust and lines of communication may optimize execution. For trauma, multiple existing structures are focused on this goal, including the Coalition for National Trauma Research.27 Combinations of networks and trial platforms may allow for rapid growth of an MCT, including a recent pandemic example of rapid execution of a multiplatform randomized controlled trial.28

An additional challenge, beyond communication, that exists in orchestrating an MCT is the navigation of regulations and contracting. These processes can vary widely between institutions, and, if international, between nations. Contracting and regulatory processes can be a major rate-limiting step in the launch of MCTs. One potential asset to increase the efficiency of these processes is the use of the NIH National Center for Advancing Translational Sciences (NCATS) Clinical and Translational Science Awards (CTSA) program. NCATS and CTSA have generated a clinical research toolbox which includes items such as an accelerated clinical trial agreement, which is a standardized contract model designed to reduce negotiations and time in contracting. This template and other harmonizing tools are available at: https://ncats.nih.gov/expertise/clinical.29

Arguably the most important aspect of MCT execution is patient enrollment. Numerous factors and important considerations exist for patient enrollment, including optimizing inclusion and exclusion criteria, enrolling a diverse and representative cohort of participants, and creating an MCT structure that allows for optimized testing within a limited cohort of potential patients. An emerging strategy to optimize the execution of MCTs and the efficiency of testing interventions is the use of platform trials. Extensively characterized elsewhere,28 30 31 a platform trial creates a perpetual infrastructure for testing multiple different interventions.28 Although not yet in use for trauma, a plan for execution of a trauma clinical trials platform has recently been proposed.32 Alongside these innovations in trial design, the statistical plan for analysis in trauma trials is the subject of major innovation.33 The evolving popularity of Bayesian analyses in clinical trials requires unique expertise and multidisciplinary collaboration in trial design.

Overview

Article metrics
Altmetricopen-url
Dimensionsopen-url