Original Research••

Leveraging a health information exchange to examine the accuracy of self-report emergency department utilization data among hospitalized injury survivors

Abstract

Background Accurate acute care medical utilization history is an important outcome for clinicians and investigators concerned with improving trauma center care. The objective of this study was to examine the accuracy of self-report emergency department (ED) utilization compared with utilization obtained from the Emergency Department Information Exchange (EDIE) in admitted trauma surgery patients with comorbid mental health and substance use problems.

Methods This is a retrospective cohort study of 169 injured patients admitted to the University of Washington’s Harborview Level I Trauma Center. Patients had high levels of post-traumatic stress disorder and depressive symptoms, suicidal ideation and alcohol comorbidity. The investigation used EDIE, a novel health technology tool that collects information at the time a patient checks into any ED in Washington and other US states. Patterns of EDIE-documented visits were described, and the accuracy of injured patients’ self-report visits was compared with EDIE-recorded visits during the course of the 12 months prior to the index trauma center admission.

Results Overall, 45% of the sample (n=76) inaccurately recalled their ED visits during the past year, with 36 participants (21%) reporting less ED visits than EDIE indicated and 40 (24%) reporting more ED visits than EDIE indicated. Patients with histories of alcohol use problems and major psychiatric illness were more likely to either under-report or over-report ED health service use.

Discussion Nearly half of all patients were unable to accurately recall ED visits in the previous 12 months compared with EDIE, with almost one-quarter of patients demonstrating high levels of disagreement. The improved accuracy and ease of use when compared with self-report make EDIE an important tool for both clinical and pragmatic trial longitudinal outcome assessments. Orchestrated investigative and policy efforts could further examine the benefits of introducing EDIE and other information exchanges into routine acute care clinical workflows.

Level of evidence II/III.

Trial registration number ClinicalTrials.gov NCT02274688.

Introduction

Increasingly, trauma centers are being encouraged to collect self-report outcome data from patients to capture key longitudinal injury outcomes.1–3 The American College of Surgeons Committee on Trauma may recommend the routine collection of outcomes, including physical and mental health symptoms, functional outcomes, and patterns of health service utilization, during the course of the weeks and months after injury. A series of large-scale trauma center prospective cohort and clinical trial investigations have relied exclusively on patient self-report to ascertain rates of physical and mental health symptoms, functional outcomes, and patterns of health service utilization.4–6 Increasingly, large-scale pragmatic clinical trials are being implemented in acute care medical settings and with injury survivors.7–9 Pragmatic clinical trials are designed to harness data collected in clinical settings under conditions of routine practice and encourage the use of real-time, workflow-integrated outcome data collection in the conduct of comparative effectiveness research.10 11

Leveraging large-scale electronic medical data collection is therefore an increasingly essential element of both acute care clinical and pragmatic trial longitudinal outcome data assessments.1 10–12 Health information exchanges represent one potential and currently underused source of large-scale electronic medical record (EMR) data in acute care medical settings.1 12–16 A key initial step in integrating health information exchanges into routine trauma center clinical and research longitudinal outcome assessments is understanding the accuracy of self-report health service utilization in comparison to real-time, workflow-integrated administrative data sources.

However, literature review revealed few studies that have compared self-report outcomes versus administrative data from a health information exchange. Specifically, one study examined acute care utilization in hospitalized patients with substance use disorders and documented consistent under-reporting of visits when comparing patient self-reports to objective emergency department (ED) exchange data sources.17 Similarly, in other areas of health services research, self-report utilization measures have been associated with recall biases,18 and it appears that error in recall increases as utilization increases.19 20 Additionally, previous studies have investigated acute care patterns of service utilization with administrative data in cohorts of patients at risk for high utilization.21 22 However, none of these studies are of trauma patients and many of these studies are limited to single health systems,23 whereas other studies include only a small geography that may cross health systems22 24 or are limited to one specific condition.17 25 To date, no studies substantiate the utility of administrative data from health information exchanges in documenting ED utilization patterns across broad patient populations and state-wide geographic regions, despite reliance on this measure to determine effective clinical outcomes for trauma surgery patients.

The Emergency Department Information Exchange (EDIE) is a type of health information exchange or care coordination platform that collects EMR information14 26 at the time a patient checks into any ED. Patients do not need to opt in or request their information be shared; rather, hospitals decide to participate for their entire population. For any patient, a hospital follows a standard registration process, and an admission, discharge and/or transfer message containing patient demographics is delivered to Collective Medical, which then packages health information on the patient and sends a notification to treating providers in the ED.27 The information package, or EDIE ‘alert’, generated at each ED visit includes counts of prior ED visits during the past 12 months and specific EDs visited, which is obtained through EMR data from each participating hospital. EDIE is unique for an intraoperability platform given its reach and spread across the country; as of June 2020, EDIE has participation in 99% of EDs in Washington and Oregon, as well as many sites in California and over 20 other states across the country.28

The objective of this article was to first assess the patterns of past 12-month state-wide ED utilization among a cohort of injured patients admitted to a level I trauma center with comorbid medical, psychiatric and substance use conditions using EDIE. Next, the investigation compared self-report ED utilization with EDIE-documented ED visits to assess patterns of patient over-reporting and under-reporting. Finally, the investigation aimed to determine if there were patient demographics or clinical characteristics associated with the over-reporting or under-reporting of ED utilization.

Methods

Study design

This is a secondary data analysis from the screening and baseline assessment phase of a comparative effectiveness trial of patient-centered care transitions for patients with injuries and elevated risk for post-traumatic stress disorder (PTSD).29

Study setting and population

Participants were recruited from Harborview Medical Center (HMC), a large urban level I trauma center located in Seattle, Washington; HMC is the only level I trauma center for the five-state (Washington, Wyoming, Alaska, Montana and Idaho) region. Participants were recruited by trained research assistants (RAs) Monday through Friday from 07:00 to 18:00, including a sampling of evenings and weekends from March 2014 through September 2015. RAs reviewed the EMR of patients aged 14 years or older presenting to the ED and admitted for injury care for at least 24 hours to find those at elevated risk for PTSD based on a validated 10-item population-based EMR prescreen.30 Those with elevated risk for PTSD based on the EMR prescreen were approached to participate. Participants were excluded if they required immediate psychiatric care (eg, suicide attempt survivors), did not live in Washington state, were incarcerated, did not speak English, did not have a Glasgow Coma Scale score of 15 and a score of at least 7/10 on the Mini-Mental Status Exam. Adult participants provided written informed consent, and parental consent was obtained for adolescent participants with subsequent adolescent assent. Differences in demographic and injury characteristics among patients enrolled in the trial compared with all other patients admitted to HMC during the time period of the study have previously been described.29 On average, patients in the trial were approached to participate at 6.2 days after admission (SD 4.9 days).

Measurements

Mental health and substance use screening

The RAs asked participants about their postinjury concerns with the open-ended question, ‘Of everything that has happened to you since you were injured, what concerns you the most?’, allowing for an unlimited amount of concerns. Enrolled participants were required to have at least three post-traumatic concerns at the time of recruitment.31 All participants were screened for mental health comorbidity. Specifically, an RA-assisted interview was performed including the PTSD CheckList-Civilian Version,32 which is a validated 17-item screen for PTSD since the injury for which they were being treated in the hospital. A score of 35 or greater on the PTSD Checklist represents an elevated risk for PTSD,30 and the measure has been used previously in the acute postinjury phase to determine patients at risk for PTSD.6 33 Depression in the previous 2 weeks was assessed with the validated Patient Health Questionnaire 9 (PHQ-9).34 A score of 10 or higher on the PHQ-9 is sensitive for moderate depression, and the PHQ-9 has previously been used in ED and acute care settings.35 Participants with elevated risk for PTSD based on a score of ≥35 on the PTSD Checklist, or with moderate depression based on a score of ≥10 on the PHQ-9, or with suicidal thoughts based on a score of ≥1 on the suicide assessment item within the PHQ-9 were included and completed the baseline interview. Patients were also screened for alcohol use problems with the three-item Alcohol Use Disorders Identification Test (AUDIT-C)36 to assess problem drinking in the 12 months prior to injury. A score of ≥4 for males and ≥3 for females suggests alcohol misuse in the year prior to injury. The AUDIT-C is a validated instrument and has been previously used in the ED and other medical settings. History of alcohol use and history of major psychiatric illness were abstracted from the 2014–2015 trauma registry. Specifically, ‘previous history of major psychiatric illness’ includes any documented International Classification of Diseases (ICD-9) code within the HMC EMR of preinjury major depressive disorder, bipolar disorder, schizophrenia, anxiety/panic disorder, borderline or antisocial personality disorder and/or adjustment disorder or PTSD. ‘Previous history of alcohol use’ includes evidence of chronic use within the HMC EMR, including documentation of ICD-9 codes pertinent to chronic use such as alcohol withdrawal. This variable did not include toxicology information from the acute visit.

Baseline measures

Demographics

Demographic questions including age, sex and race were asked as part of the baseline interview. A single item was used to assess living situation and answers were dichotomized to ‘homeless’ (eg, homeless, shelter or temporary residence) or ‘not homeless’ (eg, home, apartment, group home or skilled nursing home). Insurance status was abstracted from the trauma registry.

Clinical characteristics

Injury Severity Score (ISS) at baseline was determined from ICD-9 codes and intensive care unit stay was determined from the trauma registry. Patients were asked to report medications taken in the 3 months before their injury hospitalization. All recorded opioid pain medications, which included hydrocodone, tramadol, oxycodone, methadone and fentanyl, were collapsed to form a variable of ‘opioid pain medication at baseline’.

ED utilization

Participants were asked to report their ED utilization in the year prior to their injury. Specifically, participants were asked, ‘During the past year prior to your injury, how many visits did you make to a hospital emergency room?’ to determine their self-report ED utilization during the 12 months prior to their injury. Information from EDIE26 was obtained to determine ED utilization within Washington and Oregon in the same time period. From EDIE, we determined the number of ED visits to any ED in Oregon and Washington for the 12 months prior to injury, as well as the number of different EDs visited by each patient during the same period.

Analysis

Descriptive statistics for demographic, clinical and ED utilization variables were calculated for all patients included in the cohort. Participants with incomplete self-report ED utilization data (n=2) were excluded from the sample. Next, the study documented and compared patterns of EDIE and self-report ED utilization including a quantitative description of amount of discordance. Linear regression was used to assess for a significant association between EDIE-documented ED visits and number of distinct hospitals visited as documented by EDIE. A Bland-Altman analysis37 was performed to assess the degree of agreement between self-report ED visits and ED visits documented by EDIE; for the Bland-Altman analysis, an absolute cut-off of >1 visit was used to define clinically relevant disagreement when comparing self-report to EDIE-documented utilization.37 38

To assess the hypothesis that patients who misreport ED visits would be more likely to have psychiatric and substance use problems, we evaluated participant ability to accurately self-report ED visits. For this analysis, self-report visits were compared with the visit history obtained from the health information exchange, or the EDIE-documented ED visits. These analyses demonstrated three categories of visit comparisons: participants were identified as ‘matching’ if their self-report value matched the EDIE value, ‘under-reporting’ if their self-report value was lower than the EDIE value and ‘over-reporting’ if their self-report value was greater than the EDIE value. First, we compared those with matching self-report and EDIE values for ‘0’ visit in the past year to those with any other value of matching self-report and EDIE visits. Baseline group differences were examined using independent group t-tests for continuous variables and χ2 tests for categorical variables. No differences were found between these groups on any variables, and therefore these groups were combined to form the ‘matching’ group. Next, we used one-way analysis of variance for continuous variables and χ2 tests for categorical variables to identify differences across the three groups in demographics, mental health comorbidity, clinical characteristics and ED visit utilization among the matching, under-reporting and over-reporting groups. Planned post hoc differences were examined using Bonferroni comparisons. Analyses were performed using SPSS V.24.

Results

A total of 169 patients with trauma and elevated risk for mental health and substance use problems were included in the cohort (table 1). The average age was 42.1 years (SD 15.9 years), 56.8% (n=96) were female and approximately half identified as white (55.6%). The majority had public insurance (n=163, 96.4%) and 8.3% (n=14) were homeless. Most of the sample endorsed moderate depression in the past 2 weeks (n=155, 91.7%) or risk for PTSD (n=127, 75.1%) whereas a quarter had some degree of suicidal thoughts (n=41, 24.3%) which is consistent with the recruitment effort of the trial.29 Additionally, half (n=83, 49.1%) had past year alcohol misuse and 13.6% (n=23) had a record of chronic alcohol use documented in the EMR.

Table 1
|
Demographics, mental health and substance use, clinical characteristics and emergency department (ED) visit utilization among trauma patients with elevated risk for mental health and substance use comorbidity (n=169)

During the year prior to the index admission, the average number of self-report ED visits was 1.3 (SD 2.4) with EDIE showing an average of 1.5 (SD 2.5) ED visits in the past 12 months (table 1). By the self-report measure, 51.5% (n=88) of participants reported no past year ED visits, 18.9% (n=32) reported one ED visit, 22.5% (n=38) reported two to four ED visits and 6.5% (n=11) reported five or more ED visits in the past year. Evaluation of EDIE data for past year visits revealed 56.0% (n=96) of participants had no ED visits, 12.9% (n=22) had one ED visit, 19.3% had two to four ED visits and 11.7% (n=20) had five or more ED visits.

Comparing these reporting methods, 55% (n=93) matched with self-report and EDIE methods providing the same value, 9.9% (n=17) under-reported with EDIE having a value of exactly 1 more than self-report and 13.5% (n=23) under-reported with EDIE having a value of 2 or more higher than self-report. A total of 11.7% (n=20) reported more visits than EDIE by exactly 1 and 9.5% (n=16) had a self-report value of 2 or more higher than EDIE. Overall, the self-reports of 23% (n=39) of participants differed by the clinically relevant absolute cut-off of >1 visit when compared with EDIE-documented utilization (figure 1).

Figure 1
Request permissionrequest permission icon
Figure 1

Bland-Altman plot of differences between participants’ self-report and EDIE-documented number of emergency department (ED) visits versus the mean number of ED visits. EDIE, Emergency Department Information Exchange.

By tracking both the number of EDs visited and the number of distinct hospital EDs visited, EDIE facilitated an assessment of the association between these two indices of acute care health service utilization. Linear regression was used to predict the total number of ED visits in the year prior by EDIE from the number of distinct EDs visited in the same time period, and as the number of different EDs visited increased, the number of total ED visits increased linearly (R2=0.671).

Next, the ‘matching’ group was further examined. While half (n=93, 55%) of the total sample had matching self-report ED visits and EDIE-documented ED visits during the past year (ie, matching), the majority (n=75, 80.1%) of those had zero ED visit. Within this matching group, those with matching self-report and EDIE for one or more ED visits in the past 12 months (n=18) were compared with those with zero visit (n=75) to determine if any significant differences existed between these groups. There were no statistically significant differences among demographics, current mental health characteristics, or clinical characteristics. Therefore, these two subgroups were combined (n=93) to form the ‘All Matching’ group for further analysis (table 2). In the ‘Under-reporting’ group, the mean number of self-report visits was 1.6 (SD 1.7) compared with the EDIE-documented ED visits of 4.4 (SD 3.0), and these EDIE-documented visits occurred at 1.9 different hospitals (SD 1.0). In the ‘Over-reporting’ group, the mean number of self-report visits was 3.5 (SD 3.8) compared with the EDIE-documented ED visits of 1.2 (2.4), and these visits occurred at 0.6 different hospitals (SD 0.65). Among those in the ‘Matching emergency department visits’ group with at least one ED visit in the past 12 months (n=18), the mean number of ED visits was 1.6 (SD 0.7), with these visits occurring at 1.1 different hospitals (SD 0.32).

Table 2
|
Demographics, clinical characteristics and emergency department (ED) visit utilization reporting patterns (n=169)

Baseline variables were compared among the ‘All Matching’, ‘Over-reporting’ and ‘Under-reporting’ groups (table 2). Among demographic characteristics, there were no statistically significant differences with the exception of homelessness. While the frequency of homelessness was low among the whole cohort (n=14), there were differences across the three groups, with the ‘Under-reporting’ group having more homelessness than the ‘All Matching’ or ‘Over-reporting’ groups. Participants in all groups had high rates of current mental health characteristics, and no differences existed across groups for current risk for PTSD, current major depression or suicidality. However, significant differences existed between the groups for history of alcohol use problems and history of major psychiatric illness as determined by the EMR. Post hoc comparisons revealed the ‘All Matching’ group to have less alcohol use than the ‘Under-reporting’ group and less psychiatric disease than the ‘Under-reporting’ group.

Discussion

Using data derived from EDIE for a cohort of 169 injured trauma survivors to describe patterns of health service utilization is feasible, and highlights the difficulty in accuracy of self-report in patients with complex medical, psychiatric and substance-related comorbidity admitted to a single level I trauma center in the Pacific Northwest Region of the USA. Approximately half (45%) of the injured patients with injury, mental illness and substance use comorbidity were unable to accurately report their ED utilization within the past year when compared with EDIE, a health information exchange or intraoperability platform gold standard. Additionally, just under one-quarter of patients had self-report ED use that was more than one visit over or under the EDIE gold standard. Among those that could not accurately recall or report their ED visits, approximately half under-reported their visits and half over-reported their visits. Those that under-reported ED visits were more likely to have a history of alcohol use, a history of major psychiatric illness or report homelessness compared with those that correctly recalled their ED visits. Prior investigations in hospitalized patients with substance use disorders also document inaccuracies in self-report acute care utilization.17 However, unlike the current study, substance users’ self-report errors were unidirectional, with only significant under-reporting of ED utilization.

Prospective cohort and population-based epidemiologic studies suggest that a subgroup of injured trauma survivors will develop a chronic condition characterized by multiple comorbid diagnoses including mental health, substance use and medical disorders, as well as suicide attempts and violence-related life events such as firearm injuries.4 39–41 Trauma centers are developing longitudinal outcome assessment procedures, as well as pragmatic screening and intervention procedures and policy to address the complex needs of injured patients with substance use and mental health comorbidity.1 9 42 Essential to these efforts are assessments of acute care health service utilization patterns that realistically account for potential data inaccuracies.16 43 This current investigation demonstrates the value of EDIE, one particular health information exchange, for providing a more accurate ED utilization history than can be obtained by self-report in a patient population with multiple comorbidities. This study demonstrates the ability for more accurate assessment of care through the frequently used metric of recidivistic visits in a population at risk for ED/trauma center recidivism.44 As ED visits increase, the risk for inaccurate self-report increases as well. Patients at risk for repeat ED visits, such as those with substance use and mental health comorbidity, often require trauma center care coordination.29 Likewise, developing interventions for this population requires an accurate understanding of healthcare utilization.42

Additionally, when considering recommendations for routine longitudinal outcome research from trauma centers,1 4 real-time, workflow-integrated administrative data such as EDIE provide the unique advantage of ≥90% follow-up without relying on patient-reported outcomes and interviews. Also, utilization outcome data obtained through EMRs are often limited to one health system, and many patients will visit at least two different EDs in the course of a year,22 45 46 making health information exchange systems such as EDIE an important source of these data. EDIE is unique for an intraoperability platform given its density and implementation around the country.

Health information exchanges have potential to affect trauma center clinical practice and outcomes assessment above and beyond accurate visit reporting. In addition to accurate utilization data for clinicians and outcomes researchers, health information exchanges have the potential to assess clinical quality measures,13 47 48 although this potential has yet to be used. Further, this information has potential to serve as an invaluable repository of data for health services researchers aiming to trauma center health services utilization. Important challenges to implementation of health information exchange platforms in clinical care and research include adoption and functionality. Information that is difficult to access and requires separate login and interruption of clinical workflow can lead to low usability.49 Clinical investigators working in acute care settings can help evaluate health information exchanges or intraoperability platforms such as EDIE, but need to design studies and consider issues such as patient privacy and data sharing to collect this type of information.

Limitations

Despite these novel observations, there are some important limitations to note. First, all participants included in this study were recruited from a single site, and results may not generalize to other types of hospitals. Additionally, the sample size was fixed, as this was a secondary analysis of a randomized clinical trial and certain groups were small, limiting the power to detect significant differences. Though participants had to report residency in Washington to be included, it is possible they could have recently moved into the region, limiting data availability from EDIE for the 12 months prior. This would inflate the number of participants we found to be over-reporting in our sample. The self-report recall period was 1 year in duration, and it is known that shorter recall times result in improved accuracy of self-report.18 All health information exchanges, including EDIE, are imperfect systems with internal limitations, specifically related to matching and data accuracy, although previous research demonstrated current matching algorithms using minimal identifying data such as first name, last name and date of birth are extremely accurate.50 Patients unable to provide this information at the time of check-in would be missed in any health information exchange.

Conclusion

Emergency Department Health Information Exchange administrative data can provide more accurate information regarding patterns of ED utilization in patients with high levels of medical, mental health and substance-related comorbidity than self-report. As a prospective outcome assessment tool, ED health information exchanges such as EDIE have the advantages of enhanced accuracy and ease of longitudinal follow-up when compared with self-report. Orchestrated investigative and policy efforts could further examine the benefits of introducing EDIE and other information exchanges into routine trauma center and emergency care workflows.

Overview

Article metrics
Altmetricopen-url
Dimensionsopen-url