Clinical surveillance of sepsis. Decade of sepsis sniffer.
by Vitaly Herasevich, MD, PhD, MSc Ambient Clinical CMIO
Sepsis continues to be a worldwide problem. Recently, there was a large study published in the Journal of Critical Care that found 2.5 million people have been diagnosed with Sepsis An important point that was emphasized in this study was that the cost of a patient who has sepsis present at admission ($18,023) varies widely from a patient who does not have sepsis present at admission ($51,022) [1]. These numbers demonstrate the clear importance of early in-hospital sepsis identification. As a syndrome, sepsis and septic shock were defined based on SIRS in 1992 at the consensus conference, with some later changes [2]. The most recent Sepsis-3, qSOFA based definition confirmed the importance of physiological abnormalities as foundation of sepsis and septic shock. This JAMA paper defined “septic shock as a subset of sepsis in which underlying circulatory, cellular, and metabolic abnormalities are associated with a greater risk of mortality than sepsis alone. Adult patients with septic shock can be identified using the clinical criteria of hypotension requiring vasopressor therapy to maintain mean BP 65 mm Hg or greater and having a serum lactate level greater than 2 mmol/L after adequate fluid resuscitation” [3]. Later, authors published commentary stating “qSOFA does not replace SIRS in the definition of sepsis” where they stated “We hope this editorial will clarify that the qSOFA is meant to be used to raise suspicion of sepsis and prompt further action—it is not a replacement for SIRS and is not part of the definition of sepsis” [4].
What progress has been made over the years in electronic sepsis surveillance or sepsis “sniffing”? Taking a look over the years since 2005, very little has been done due to the absence of electronic data and EMR availability. One of the first reports on sepsis electronic clinical surveillance was published in 2008 [5]. The system showed suboptimal diagnostic performance in a 2-month implementation trial. Since then, multiple publications depicted improved accuracy, however, no available solution proved to be universally applicable. As modern statistical approach methods try to catch up with sensitivity and specificity of the detection mode, the reality is that an electronic alert by itself cannot solve the problem of poor sepsis outcomes. A SIRS based algorithm in one study published in 2014 showed sensitivity of 93%, specificity of 98%, positive predictive value of 21% and negative predictive value of 99.97% for severe sepsis and septic shock [6]. Another study showed similar results with sensitivity of 80% and a specificity of 96% [7]. Later studies used different machine learning approaches but most of them had 3 significant limitations:
#1: Using ICD codes as an identification of sepsis.
#2: Most are “Database studies” – derivation and validation cohorts extracted from existing databases.
#3: Data timing. If all data is ready at time of model running there are no component of time. In real time scenarios not all data is available at time of screening.
For example, prospective validation of machine learning random-forest classifier for sepsis prediction demonstrated a sensitivity of 26% and specificity of 98%, with a positive predictive value of 29% [8]. Another machine learning study involved MIMIC Database and ICD codes as an outcome measurement used gradient tree boosting technique showed similar to traditional approach sensitivity of 80% and specificity of 84% [9].
Commercial EMR vendors deployed their own version of sepsis sniffers for clinical surveillance. A validation study of EPIC EMR sepsis sniffer used ICD codes as “gold standard” resulting in asensitivity of 75% and specificity of 86% [10]. A validation study of Cerner’s St. John Sepsis Surveillance Agent does not report diagnostic performance rater then AUC=75% that could be translated to very similar sniffer characteristics [11].
It is evident that despite improvements in modern statistical techniques and machine learning approaches, sepsis alert performance has not improved in the past decade. The biggest question is: what is the best path moving forward?
Alarm fatigue is an issue that many health systems are facing due to the increase in alerts. The answer to this problem is quite simple. Better staff education and improved embedding of sepsis surveillance tools in clinical workflows, dealing with health care delivery models, and changing charting behaviors with real-time availability of electronic data [12]. This was proven in 2011 when an improvement in aggressive sepsis management was evident after the implementation of a sepsis sniffer [13]. Literature has shown that there has been no visible change in sepsis detection/prediction since its first reports 12 years ago. Correct implementation and embedding a solid sepsis sniffer tool in clinical process is the key for success.
References
1. Paoli CJ, Reynolds MA, Sinha M, Gitlin M, Crouser E. Epidemiology and Costs of Sepsis in the United States-An Analysis Based on Timing of Diagnosis and Severity Level. Crit Care Med. 2018 Dec;46(12):1889-1897. PMID: 30048332;
2. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, Schein RM, Sibbald WJ. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992 Jun;101(6):1644-55. PMID: 1303622.
3. Shankar-Hari M, Phillips GS, Levy ML, Seymour CW, Liu VX, Deutschman CS, Angus DC, Rubenfeld GD, Singer M; Sepsis Definitions Task Force. Developing a New Definition and Assessing New Clinical Criteria for Septic Shock: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016 Feb 23;315(8):775-87. PMID: 26903336.
4. Vincent JL, Martin GS, Levy MM. qSOFA does not replace SIRS in the definition of sepsis. Crit Care. 2016 Jul 17;20(1):210. PMID: 27423462.
5. Herasevich V, Afessa B, Chute CG, Gajic O. Designing and testing computer based screening engine for severe sepsis/septic shock. AMIA Annu Symp Proc. 2008 Nov 6:966. PMID: 18999146.
6. Alsolamy S, Al Salamah M, Al Thagafi M, Al-Dorzi HM, Marini AM, Aljerian N, Al-Enezi F, Al-Hunaidi F, Mahmoud AM, Alamry A, Arabi YM. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak. 2014 Dec 5;14:105. PMID: 25476738
7. Harrison AM, Thongprayoon C, Kashyap R, Chute CG, Gajic O, Pickering BW, Herasevich V. Developing the surveillance algorithm for detection of failure to recognize and treat severe sepsis. Mayo Clin Proc. 2015 Feb;90(2):166-75. PMID: 25576199
8. Giannini HM, Ginestra JC, Chivers C, Draugelis M, Hanish A, Schweickert WD, Fuchs BD, Meadows L, Lynch M, Donnelly PJ, Pavan K, Fishman NO, Hanson CW 3rd, Umscheid CA. A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice. Crit Care Med. 2019 Nov;47(11):1485-1492. PMID: 31389839.
9. Mao Q, Jay M, Hoffman JL, Calvert J, Barton C, Shimabukuro D, Shieh L, Chettipally U, Fletcher G, Kerem Y, Zhou Y, Das R. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open. 2018 Jan 26;8(1):e017833. PMID: 29374661
10. Rolnick J, Downing NL, Shepard J, Chu W, Tam J, Wessels A, Li R, Dietrich B, Rudy M, Castaneda L, Shieh L. Validation of Test Performance and Clinical Time Zero for an Electronic Health Record Embedded Severe Sepsis Alert. Appl Clin Inform. 2016 Jun 22;7(2):560-72. PMID: 27437061
11. Amland RC, Sutariya BB. Quick Sequential [Sepsis-Related] Organ Failure Assessment (qSOFA) and St. John Sepsis Surveillance Agent to Detect Patients at Risk of Sepsis: An Observational Cohort Study. Am J Med Qual. 2018 Jan/Feb;33(1):50-57. PMID: 28693336;
12. Harrison AM, Gajic O, Pickering BW, Herasevich V. Development and Implementation of Sepsis Alert Systems. Clin Chest Med. 2016 Jun;37(2):219-29. PMID: 27229639;
13. Sawyer AM, Deal EN, Labelle AJ, Witt C, Thiel SW, Heard K, Reichley RM, Micek ST, Kollef MH. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011 Mar;39(3):469-73. PMID: 21169824.
Sepsis continues to be a worldwide problem. Recently, there was a large study published in the Journal of Critical Care that found 2.5 million people have been diagnosed with Sepsis An important point that was emphasized in this study was that the cost of a patient who has sepsis present at admission ($18,023) varies widely from a patient who does not have sepsis present at admission ($51,022) [1]. These numbers demonstrate the clear importance of early in-hospital sepsis identification. As a syndrome, sepsis and septic shock were defined based on SIRS in 1992 at the consensus conference, with some later changes [2]. The most recent Sepsis-3, qSOFA based definition confirmed the importance of physiological abnormalities as foundation of sepsis and septic shock. This JAMA paper defined “septic shock as a subset of sepsis in which underlying circulatory, cellular, and metabolic abnormalities are associated with a greater risk of mortality than sepsis alone. Adult patients with septic shock can be identified using the clinical criteria of hypotension requiring vasopressor therapy to maintain mean BP 65 mm Hg or greater and having a serum lactate level greater than 2 mmol/L after adequate fluid resuscitation” [3]. Later, authors published commentary stating “qSOFA does not replace SIRS in the definition of sepsis” where they stated “We hope this editorial will clarify that the qSOFA is meant to be used to raise suspicion of sepsis and prompt further action—it is not a replacement for SIRS and is not part of the definition of sepsis” [4].
What progress has been made over the years in electronic sepsis surveillance or sepsis “sniffing”? Taking a look over the years since 2005, very little has been done due to the absence of electronic data and EMR availability. One of the first reports on sepsis electronic clinical surveillance was published in 2008 [5]. The system showed suboptimal diagnostic performance in a 2-month implementation trial. Since then, multiple publications depicted improved accuracy, however, no available solution proved to be universally applicable. As modern statistical approach methods try to catch up with sensitivity and specificity of the detection mode, the reality is that an electronic alert by itself cannot solve the problem of poor sepsis outcomes. A SIRS based algorithm in one study published in 2014 showed sensitivity of 93%, specificity of 98%, positive predictive value of 21% and negative predictive value of 99.97% for severe sepsis and septic shock [6]. Another study showed similar results with sensitivity of 80% and a specificity of 96% [7]. Later studies used different machine learning approaches but most of them had 3 significant limitations:
#1: Using ICD codes as an identification of sepsis.
#2: Most are “Database studies” – derivation and validation cohorts extracted from existing databases.
#3: Data timing. If all data is ready at time of model running there are no component of time. In real time scenarios not all data is available at time of screening.
For example, prospective validation of machine learning random-forest classifier for sepsis prediction demonstrated a sensitivity of 26% and specificity of 98%, with a positive predictive value of 29% [8]. Another machine learning study involved MIMIC Database and ICD codes as an outcome measurement used gradient tree boosting technique showed similar to traditional approach sensitivity of 80% and specificity of 84% [9].
Commercial EMR vendors deployed their own version of sepsis sniffers for clinical surveillance. A validation study of EPIC EMR sepsis sniffer used ICD codes as “gold standard” resulting in asensitivity of 75% and specificity of 86% [10]. A validation study of Cerner’s St. John Sepsis Surveillance Agent does not report diagnostic performance rater then AUC=75% that could be translated to very similar sniffer characteristics [11].
It is evident that despite improvements in modern statistical techniques and machine learning approaches, sepsis alert performance has not improved in the past decade. The biggest question is: what is the best path moving forward?
Alarm fatigue is an issue that many health systems are facing due to the increase in alerts. The answer to this problem is quite simple. Better staff education and improved embedding of sepsis surveillance tools in clinical workflows, dealing with health care delivery models, and changing charting behaviors with real-time availability of electronic data [12]. This was proven in 2011 when an improvement in aggressive sepsis management was evident after the implementation of a sepsis sniffer [13]. Literature has shown that there has been no visible change in sepsis detection/prediction since its first reports 12 years ago. Correct implementation and embedding a solid sepsis sniffer tool in clinical process is the key for success.
References
1. Paoli CJ, Reynolds MA, Sinha M, Gitlin M, Crouser E. Epidemiology and Costs of Sepsis in the United States-An Analysis Based on Timing of Diagnosis and Severity Level. Crit Care Med. 2018 Dec;46(12):1889-1897. PMID: 30048332;
2. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, Schein RM, Sibbald WJ. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992 Jun;101(6):1644-55. PMID: 1303622.
3. Shankar-Hari M, Phillips GS, Levy ML, Seymour CW, Liu VX, Deutschman CS, Angus DC, Rubenfeld GD, Singer M; Sepsis Definitions Task Force. Developing a New Definition and Assessing New Clinical Criteria for Septic Shock: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016 Feb 23;315(8):775-87. PMID: 26903336.
4. Vincent JL, Martin GS, Levy MM. qSOFA does not replace SIRS in the definition of sepsis. Crit Care. 2016 Jul 17;20(1):210. PMID: 27423462.
5. Herasevich V, Afessa B, Chute CG, Gajic O. Designing and testing computer based screening engine for severe sepsis/septic shock. AMIA Annu Symp Proc. 2008 Nov 6:966. PMID: 18999146.
6. Alsolamy S, Al Salamah M, Al Thagafi M, Al-Dorzi HM, Marini AM, Aljerian N, Al-Enezi F, Al-Hunaidi F, Mahmoud AM, Alamry A, Arabi YM. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak. 2014 Dec 5;14:105. PMID: 25476738
7. Harrison AM, Thongprayoon C, Kashyap R, Chute CG, Gajic O, Pickering BW, Herasevich V. Developing the surveillance algorithm for detection of failure to recognize and treat severe sepsis. Mayo Clin Proc. 2015 Feb;90(2):166-75. PMID: 25576199
8. Giannini HM, Ginestra JC, Chivers C, Draugelis M, Hanish A, Schweickert WD, Fuchs BD, Meadows L, Lynch M, Donnelly PJ, Pavan K, Fishman NO, Hanson CW 3rd, Umscheid CA. A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice. Crit Care Med. 2019 Nov;47(11):1485-1492. PMID: 31389839.
9. Mao Q, Jay M, Hoffman JL, Calvert J, Barton C, Shimabukuro D, Shieh L, Chettipally U, Fletcher G, Kerem Y, Zhou Y, Das R. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open. 2018 Jan 26;8(1):e017833. PMID: 29374661
10. Rolnick J, Downing NL, Shepard J, Chu W, Tam J, Wessels A, Li R, Dietrich B, Rudy M, Castaneda L, Shieh L. Validation of Test Performance and Clinical Time Zero for an Electronic Health Record Embedded Severe Sepsis Alert. Appl Clin Inform. 2016 Jun 22;7(2):560-72. PMID: 27437061
11. Amland RC, Sutariya BB. Quick Sequential [Sepsis-Related] Organ Failure Assessment (qSOFA) and St. John Sepsis Surveillance Agent to Detect Patients at Risk of Sepsis: An Observational Cohort Study. Am J Med Qual. 2018 Jan/Feb;33(1):50-57. PMID: 28693336;
12. Harrison AM, Gajic O, Pickering BW, Herasevich V. Development and Implementation of Sepsis Alert Systems. Clin Chest Med. 2016 Jun;37(2):219-29. PMID: 27229639;
13. Sawyer AM, Deal EN, Labelle AJ, Witt C, Thiel SW, Heard K, Reichley RM, Micek ST, Kollef MH. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011 Mar;39(3):469-73. PMID: 21169824.