Top Ten Characteristics of a Software Safety Culture

This is a follow-on to my previous post regarding the safety issues of EHRs…. But first, one comment and point of emphasis before I dive into the Top Ten details: Tightly-integrated, monolithic EHR solutions that rely on a single data model are much less prone to safety risk scenarios than those associated with loosely integrated, best-of-breed HL7-based systems. That’s my personal observation-- no lengthy study required to prove it to me. There are, of course, other drawbacks to purchasing a monolithic EHR from a single vendor, but from a safety standpoint, they are a better choice than buying disparate pieces and trying to glue them together with HL7. The unfortunate downside to HL7 is its fragility—errors are frequent and occur easily-- leading to mismatched patient records, delayed delivery of vital results, and lost delivery of the same. I’m not a fan of message oriented architectures (MOA)—never have been-- but especially so in healthcare where HL7 has made it insidiously easy to hang onto old school, unreliable MOA designs.

Now on to the Top Ten…

My information systems and software development background started in the U.S. Air Force, an environment in which software safety (and security) is an embedded part of the culture. Military weapons control and targeting systems, flight control systems, and intelligence data processing systems are all developed with an inherent sense of safety. That background predisposed me towards an interest in the subject, especially so as I watched software and firmware play a larger and larger role in those areas of our lives that were once controlled by levers, gears and links. I have a gloomy sense of self-satisfaction in knowing that I predicted-- and tried on numerous occasions to affect the prevention of—the software system failures that affect many of our computerized voting systems and now seem also to affect our automotive control systems. I find myself in yet another Groundhog Day moment with this topic and EHRs.

As described by Debra Herrmann in “Software Safety and Reliability”, software is safe if… (I added the points of emphasis):

1. It has features and procedures which ensure that it performs predictably under normal and abnormal conditions

2. The likelihood of an undesirable event occurring in the execution of that software is minimized

3. If an undesirable event does occur, the consequences are controlled and contained

Below are the Top Ten Characteristics of a Safe Software Environment, at the CIO-level, based upon my 20+ years of watching and studying this topic. There are numerous lower level programming techniques that I might cover later, but these are the top characteristics that a healthcare CIO can and should directly lead. If you were a Software Safety Auditor for ONCHIT, CCHIT, HHS, or the FDA, you could use this checklist as a preliminary assessment of an EHR vendor or healthcare organization undergoing an EHR implementation.

1. Does the organization have an internal committee to whom software safety concerns and events can be escalated and reported?

2. Is there an Information Systems Safety Event Response process documented and well-known in the culture?

3. Is there a classification and metrics system in-use for categorizing and measuring the events and risks associated with software safety? Such as:

Type 1: Catastrophic. Patient life is in grave danger. The probability for humans to recognize and intervene to mitigate this event is very low or non-existent. Intervention is required within seconds to prevent the loss of life or major body function.

Type 2: Severe. Patient health is in immediate danger. The probability for humans to recognize and intervene to mitigate this event is low, but possible. Intervention is required within minutes to prevent serious injury or degradation of patient health that could lead to the loss of life or major body function.

Type 3: Moderate. Patient health is at risk. However, the probability for humans to recognize and intervene to mitigate this event is probable. Intervention is required within hours or a few days to prevent a moderate degradation in patient health.

Type 4: Minor. Patient health is minimally at risk. The probability for humans to recognize and intervene to mitigate this event is high. Corrective action should occur within days or weeks to avoid any degradation in patient health.

4. Are software applications, functions, objects, and system interfaces classified and tested according to their potential impact on safety risk scenarios? Such as:

a. Data which appears timely and accurate, but is not
b. Data that is not posted—incomplete record
c. Valid data that is accidentally deleted
d. Data posted to the wrong patient record
e. Errors in computerized protocols and decision support tools and reports

5. Do software developers follow independent validation and verification (IV&V) of their safety-critical software?

6. Is there a process of certification for programmers who are authorized to work on safety-critical software?

7. Are fault tree and failure modes effects analysis an embedded element of the software development and software safety culture?

8. Is scenario-based software testing utilized?

9. Are abnormal conditions and/or inverse requirements testing performed?

10. Is there a more risk-averse software change control and configuration management process in-place for safety-critical software?

In sharing these Top Ten Characteristics, based upon my experience, it is an attempt to contribute to the solution, not just complain about the problem. If you have any thoughts, suggestions, or personal lessons, I’d enjoy hearing from you.

Displaying the Status of the EHR vs. The Patient

A CIO colleague recently shared with me that her EHR team runs a routine that balances the clinical results that are sent out against those that are posted to the EHR. I gather from her message that the results of that reconciliation are used by her team to manage the environment, but not shared with clinicians in some sort of a data quality dashboard.

In the rest of our lives, there are examples everywhere in which we display the status of the system to the operator-- automobiles, pilots, manufacturing systems, transportation system, et al. In healthcare we display the status of the patient via the EHR, but we don't display the status of the EHR to the operator-- that's the system status that's missing in this healthcare process-- the status of the data quality (completeness and validity) in the EHR is the unknown that leaves clinicians not knowing what they don't know.

We need some creative ways to display the status of the EHR's data quality to clinicians, such as:

--Average delay in posting results to the EHR
--Likelihood that "this patient" is a duplicate record
--Average daily backlog of HL7 messages per patient in the master patient index

Patient Safety and Electronic Health Records

Below is a discussion motivated by four sentinel events which were root-cause attributable to electronic health records (EHRs) for which my teams and I were personally responsible.

Remember when safety belts in automobiles first became popular? They were simple lap belts, no shoulder strap. Did they aid passenger safety? Yes, in some ways… but they also introduced the danger of a whole new range of injuries, such as lumbar separation and paralysis, which hadn’t previously existed. It wasn’t until we added shoulder straps and the three point anchor to seat belts in cars that the evidence of benefit to passenger safety became clear and without question. We need pause now and add shoulder straps to EHRs.

Recent discussions at the national level about the patient safety effects of electronic health records are encouraging. The Wall Street Journal, Huffington Post, ONCHIT, Congress, and the FDA are starting to pay attention. But I laughed sadly and out loud when I read that the FDA estimated 44 patient deaths in the last two years “might be” attributable to the use of EHRs. That number is lowly inaccurate and also ignores all the other undesirable patient safety risk scenarios, besides death, that deserve attention.

Speaking from 13 years in the trenches of EHR-use at a variety of healthcare organizations, and applying a dash of country-boy common sense, I don’t need a fancy and sophisticated, multi-year analysis to tell me that the impact is orders of magnitude higher—it’s 100 times higher, at least. There are unintended consequences of EHRs upon patient safety every day in every hospital that uses an EHR.

Here’s a brief discussion of only four of the many patient safety events that happened under my watch, that were directly attributable to an EHR.

Missed diagnosis for soft tissue sarcoma: In this accident, a soft tissue sarcoma went undiagnosed for at least 3 months, possibly as long as 6 months, because the radiologist’s report from the Radiology Information System failed to file properly in the EHR and the referring physician didn’t know what they didn’t know and failed to follow-up on the results of the scan. The young mother of three went untreated, the cancer spread to the point that it was untreatable, and she died. The case was settled out of court for several hundred thousand dollars.

Acute Renal Failure from lack of lab results: The patient was administered a contrast dye in preparation for an imaging procedure, even though a recent lab test revealed compromised kidney function. The lab result failed to file properly in the EHR and the ordering physician didn’t know what they didn’t know. The patient suffered Acute Renal Failure and the case was quietly settled out of court for several hundred thousand dollars.

Missed and inappropriate diagnosis for Congestive Heart Failure: The echo report from the cardiologist clearly indicated that the patient was suffering from a very low LVEF, but the report did not file to the correct patient record—it filed to the wrong patient record. The correct patient went untreated and eventually died. The incorrect patient received aggressive medication treatment for CHF when that treatment was unnecessary. The case was quietly settled out of court. The CHF patient’s family was given a seven figure settlement, with a requirement to keep the incident quiet. The other patient’s family was given an apology.

Infant death: A young, single and non-English speaking mother was in hard labor and her baby under duress, but the Application Programming Interface (API) failed which had been developed to capture contraction waveforms and fetal heart data and then display those for remote viewing in the EHR. This software error also caused the fetal monitor to stop generating local audible alarms. The nurses were left unaware of the patients’ condition and the treating OB/GYN, who was monitoring the patient from his home via the patient’s EHR, was left unaware. The young and frightened mother continued quietly through her labor, giving birth in the hospital bed, unattended. Her baby was born brain-dead and later died due to umbilical strangulation.

There is reasonable but not widespread or entirely tangible evidence that EHRs can reduce patient safety events — drug-drug interactions being the most obvious — though if you’ve seen the way the vast majority of drug-drug interaction alerts function in EHRs, those alerts are of little value to physicians the vast majority of the time and are generally ignored.

Though EHRs might reduce some patient safety events, I’ve seen first-hand that the use of EHRs can also introduce new categories and types of events that we cannot ignore in the rush to deploy them across the United States. We have enough experience and evidence with EHRs to pause and do better with their impact on patient safety going forward. It doesn’t take a rocket scientist’s analysis to make immediate and major improvements to this environment, and we don’t need to wait until 2013 to develop a national database to simply begin ‘measuring’ the problem.

We need to implement mandatory reporting of safety-related EHR events — no more back-room, secret meetings and out-of-court settlements — what if we allowed the FAA to function in safety secrecy like that after a plane wreck? And we need to implement immediate requirements for auditable best practices in the development and deployment of EHRs, especially in the form of rigorous testing (for example, inverse requirement testing, as is common in military and nuclear power systems) and independent verification and validation of software code, including that associated with HL7 interfaces.

And we need to do it today so an electronic system is never responsible for doctors not knowing what they don’t know.

The Elimination of Performance Reviews

I've seen at least a dozen different employee performance evaluation systems in my career--all well-intended-- but they generally do more harm than good. Many companies are eliminating them altogether. At least one company decided recently to change them to "Employee Appreciation Reviews"-- only positive comments allowed.

My logic for their upheaval is, if you see yourself as a great leader and manager, then you will only have great employees on your teams. If you have something less than great employees, then that means that you are retaining people that you shouldn't-- and therefore, you are not a good leader nor manager and you shouldn't be in the job.

Therefore, if you have only great employees on your teams, then the performance review process is simply a formal way of telling people how much you appreciate them and building their self-esteem. It also provides them with a document that they can carry with them for reference if and when they decide to take their career elsewhere. Ninety-five percent of the world will rise to the level you treat them and believe in them. Using the performance review process to grade human beings as if they were beef cattle or chicken eggs is demeaning and incredibly old-school. You should either eliminate them altogether or use them to build self-esteem. If you need to give the boot to an under-performing employee, there are other better ways of handling and documenting that process-- letters of warning, etc.

This article, below, from the Wall Street Journal, summarizes a very good book on the topic and one that I endorse 100%.

Get Rid of the Performance Review!

Eureka Moments and Death of the Deskphone

I’ve been here in the Caymans for six months now. We’ve got a nicely defined 2-3 year IT Roadmap and strategic plan and we are starting to chip away at it. A recent independent assessment—sort of a mini audit—rated 12 areas of our IT state of affairs. At my request, the assessment team rated the 12 areas on a scale of 1-10 according to generally accepted best practices, with 10 being the best. We averaged 2.8 out 10, so I’m a little worried that my dreams of enjoying Mother Nature and the locale might suffer during the next 2-3 years.

We’ll see… anyway, on our Roadmap is a plan to convert to VoIP telephony in the next 9 months. The previous plan, before my arrival, was to convert to VoIP this year, but that was a bit naïve, given the incredibly decayed state of our network cabling infrastructure, and overall network design—very flat and wide. The infrastructure can barely stay afloat providing data services, and I wouldn’t want to risk voice services on top of that, too.

That sets the stage for the new play: “Death of the Desk Phone” (aka: The Evolution of a Eureka Moment)

Scene 1: Stuck in old thinking about new VoIP. Pulling out my old files, searching My Documents for old project plans, standing in front of a white board drawing familiar diagrams of the past– Planning a traditional VoIP conversion, much like what we went through at Northwestern the last couple of years.

Scene 2: Negotiating cellular discounts. Huddled around a table with vendors on one side, me on the other, each of us with notebooks and calculators. While planning the “done it before” VoIP conversion, I’m also renegotiating new rates for cell phone services which, as the phrase goes, is a race towards zero margins for the vendors, once again. Likewise, the new billing models benefit from dual SIM technology to make cost control a nicely shared accountability between the health system and the employee.

Scene 3: Never at my desk during work hours. An empty office chair, the landline telephone ringing. (If a phone rings in the forest and nobody is around to hear it, does it make a sound?) I’m running around to this meeting and that meeting so frequently, nobody can ever reach me at my office phone. They leave messages on my voicemail or with my Assistant and I try to return their calls when I can. I go so far to change the signature block on my email to read that I’m “rarely there” next to my office phone, hoping that people will instead call my cell phone so I can stop managing two voicemails.

Scene 4: No fixed-line home phone. A lonely, dusty old Panasonic wireless landline phone, wrapped and strangled in its cord, sitting in a storage box. I haven’t had a landline home phone in almost 4 years.

Scene 5: Look to the kids for a peak of the future. A college dorm, buzzing with the energy of youth. I read an article in the Chicago Tribune that lists the things that you will no longer find in a college kid’s dorm room: CD tower, ironing board, alarm clock, boom box, heating plate, and a landline telephone. The University of Illinois eliminated landline telephones this year in their dorms, and nobody noticed except the parents.

Scene 6: I finally get it (Eureka!). Late at night, all alone, staring out my dark office window at the parking lot below… then my eyebrows raise and I rub my chin thoughtfully and nod in agreement with my thoughts. The other scenes come together into my recognition that the old-school landline office desk phone is dying… and if it’s dying, why should I invest a handsome sum of money in a VoIP conversion which, for the most part, perpetuates the old-school telephone environment?

Scene 7: New VoIP Strategy. Another meeting room with vendors on one side of the table and me on the other, but now we’re all excited…laughing and cajoling about this great new strategy and the limitless possibilities of a truly mobilized and affordable healthcare system. I’m meeting with my telephony vendors to craft a strategy that will essentially turn our 800-employee public healthcare system into a mobile telephone environment, eliminating as many fixed line office phones as we possibly can and minimizing any investment which keeps the landline telephone system on life support.

Scene 8: Mobile, shiny, happy people, R.E.M. style. My feet propped out in front of me on a lounge chair, relaxing on the beach, taking care of business — Bachman Turner Overdrive style — reveling in the beauty of the mobile healthcare environment. Oh sure, we’ll still need landline phones in a few key areas… but 5 years from now, I’m betting we have 80% fewer landline office phones than we do now. Eight years from now, we’ll be completely cellular…?

So, if you are a smaller healthcare system, trying to stretch every dollar, and haven’t converted to VoIP yet, you might give this strategy some thought. And you might want to consider the universal stages of my Eureka moment. Make time for observation, reflection, and a willingness to take on reasonable risk. Nobody got fired for renewing their landline contracts and nobody dies when they take the safest route… but neither do we excel.

Blogging at a New Site

My very dear friend, Anthony Guerra, former editor of Healthcare Informatics magazine, started a new online publication. Honored to support him with an occasional editorial, I posted one last night. I'll eventually double post to this blogger.com site, too.

http://healthsystemcio.com/

Ego and Metrics Gone Awry

It's not good karma to take pot shots at someone after they pass away, but I started this blog several months ago and never finished it, so I find at least some relief in that. The core message in this post: Beware the leader too comfortable in the role, too adept at eluding the grasp of humility.

Robert McNamara died last Monday. For those interested in studying and learning about leadership and management, independent of industry or purpose, he's a character worth knowing. My father, brother, uncle, and cousin all served in the McNamara War; my father at the most senior levels of the Air Force. And, of course, in my own Air Force professional education, we studied his style, strategies, and outcomes in-depth. Bless him and the hearts of his left behind family, his life offers poignant and painful lessons in leadership that we should all note.

He had all the right academic credentials, corporate pedigree... and haircut. He was a data wonk before it was fashionable. He came into his role as Secretary of Defense with an ego only rivaled in modern times by characters such as Patton and MacArthur. He created a dream team at the Pentagon and set out to change the stodgy US military. To that end, he had remarkable vision and management skills, and his thinking and fingerprints remain to this day in the DoD. But... his great successes fed his great ego and the egos of his aides and staff. Like an engine of pride consuming itself for fuel, it was only a matter of time before it all came apart. Ironically, for truly great leaders, their successes lead to successively greater connections to gratitude and humility. They see that, were it not for the grace of God and subtle random events to their favor, their great successes would have been great disasters.

In McNamara's War, number crunching, organizational theory, "systems engineering", academic credentials, and money were all trumped by the heart and will of a culture not inclined to change from the influence of a superpower consumed by its own self-righteousness. One of McNamara's metrics-gone-wrong legacies: The Daily Body Count, mandatorily reported up-channel from the lowest level troops in the field.  Imagine being a 20-year old platoon leader, pausing after a jungle battle to count bodies--the enemy's and your own troops--and coldly radioing them into headquarters to ensure that the numbers made it to the next morning's briefing for the Secretary of Defense.  As history would prove, those metrics were no indicator of success.  As they say, not all things can be measured and not all things measurable should be measured.  McNamara's ego convinced him that his pedigree could out-think, out-manage, and out-measure the lowly peasant Communists, but such was not the case.

In public settings, McNamara held firm to his decisions and ego until his dying day...but, I have to believe that privately, he knew better.  The best leaders walk the thin line between confidence and uncertainty... between egotism and humility. You want a leader who shows evidence of balancing both and occasionally falling to both sides of the line-- too much confidence and too much humility-- because all of those behaviors are appropriate at the right time and right place.