Saturday, October 27, 2012

One Year Back In Durango


Living and working from Durango is awesome... it's great to be back home.  We are sharing time with Mom at the perfect time in her life.  Her health is amazing for 88 yrs, thus life is still fun for her and we are having fun with her, too.  But her health is definitely on the decline.  The timing of living here is perfect-- 100% confirmation of a life-changing conversation in prayer and meditation about two years ago.

But working from Durango has its challenges.  The logistics and inefficiencies of travel are insane...physically exhausting at times.  You feel disconnected from the people and teams that you would otherwise be working very closely with.  You are disconnected socially from Durango because 95% of your interactions are external to the community.  This physical disconnect also gives you a feeling of "out of sight, out of mind"... you occasionally feel very disposable.  You feel uncertain about the future-- Is working from Durango sustainable, economically?  Uncertainty in this answer makes you work even harder to o overcome the feeling of disposability and make sure that living in Durango will be successful and sustainable for everyone who depends on you.

These unusual feelings and Durango inefficiencies are piled on top of an inherently busy and complicated profession.  The changes in healthcare and information technology are both moving at light speed, so simply keeping up with those combinatorial changes is a challenge in itself.  Finally, as a member of the leadership team for a startup company, you feel very accountable for the success of a business that now employs 65 people, and ripples out to 150 of their family members.

So... it's not a 24/7 life of shiny happy faces living and working in Durango... but it's still the perfect place to be at the perfect time in life.

This sounds terribly whiney, doesn't it?  I'm enormously, endlessly lucky.  :-)



Friday, September 28, 2012

Make the Days Count

Saw a great quote yesterday that probably every one of you have already seen, but somehow escaped me: 


"Don't count the days. Make the days count." 


When I read this, I immediately thought: Family, friends, community. Personal health. Professional excellence. Stay focused. Get sh*t done.


:-) 

Thursday, September 27, 2012

Cultural Principles and Patterns in Healthcare Analytics

On Sept. 13, HealthLeaders Magazine published an excellent article with interviews from several healthcare visionaries, entitled, “Metrics that Matter”.
The comments in the article subtly underscored some very important principles and patterns affecting healthcare analytics and data warehousing, especially the manner in which analytics evolves within the culture of an organization. It is critically important for CIOs and other C-levels to understand these patterns and principles, so that they can choreograph the execution of their strategy accordingly; know when to move quickly when they see the patterns emerging; and alternatively know when to induce the pattern when it’s not emerging.
Stated otherwise, I’ve made many mistakes in my own organizations — and witnessed the same in organizations that I was advising — when I failed to anticipate these cultural patterns and/or failed to follow the principles.
Highlighted below are specific quotes from the HealthLeaders article, then a summary of the general, related pattern.
HealthLeaders:  ‘When Cleveland Clinic deployed its first dashboards eight years ago, they were “relatively basic, showing overall volume information in the organization,” he says. “It’s evolved drastically, to where now on our dashboards, we have information that’s updated as frequently as every 30 minutes.”’
  • Pattern #1:  Analytics in the organization always trends towards greater complexity.
  • Pattern #2:  Analytics in the organization always trends towards more timely updates and analysis of data; and towards more real-time decision making.
In the early days of a data warehouse, simple static reports issued on a monthly or weekly basis are sufficient, but as the culture becomes more data driven — more agile in its response to data — the demand for more timely data and more complex analysis of that data increases. What does this imply for the CIO?
Your data warehouse technology must be capable of growing and adapting accordingly, towards more complicated analysis and more timely updates. However, deploying your analytics and data warehouse technology to deliver real-time, complex data analysis before the need coincides with the culture of the organization can be a costly mistake, technically and organizationally.
Real-time data warehouses are more expensive to deploy and maintain, but can easily be designed with no additional cost. Design for real-time capability, but don’t implement it until you need it. If you provide that real-time capability before the culture can adopt it, you face a negative ROI from the technology investment. Even worse, by providing real-time data to an organization when the tempo of decision making and cultural change is still measured in weeks and months, you stand a very high likelihood of alienating data analysts, managers, and executives who are wondering why you are providing them with real-time data when they prefer slow-time data.
Every culture has its own “Mean Time To Improvement” (MTTI). Gauging that MTTI and aligning the delivery of data so it stimulates a paced reduction in the MTTI without accelerating past the cultural capabilities of the organization is critical.
Finally, as the data analysis becomes more complex, so do the tools required for that analysis. Early in the analytic maturity of an organization, simple bar and pie charts with static content are acceptable. Over time, especially in healthcare, the analysis will mature into very complex statistical process control, predictive analytics, and pattern recognition. No single data analysis and visualization tool can meet every need, nor should it for the sake of cost control.
CIOs should pace their advocacy for data analysis tools to match the analytic maturity of the organization, likewise they should avoid the common fault of assuming that one tool can meet all analytic needs in the organization.
HealthLeaders:  ‘But the real key to today’s business intelligence at an institution such as Cleveland Clinic is distributing the information gathered by the digital nervous system not just to top leadership, but to all those in the organization with a need to know.’
  • Pattern #3:  The most successful analytically-driven organizations invest at least as much time and money in creating a data-literate culture as what they invest in data warehousing technology.
Data literacy is measured by: (1) The ability to interact with data through analytic tools, e.g., Excel, and (2) The ability to turn the knowledge that is revealed through that data interaction into actionable results, within the role of the person interacting with the data.
Quite often, I see organizations invest significantly in the technology of analytics, e.g., an enterprise data warehouse, but they do nothing to address the data literacy of their organization in such a way that the culture can fully exploit the value of the technology. Early in my career, I would frequently consult and advise on the design and development of data warehouse technology, but I ignored the data literacy of the organization. It was quite common for me to revisit these organizations 2-3 years later to find that my prized technology was collecting dust, like a public library in a community that couldn’t read.
In addition to leading the technology strategy, CIOs must lead the charge in increasing the data literacy of the organization by hiring data literate employees across the organization — this includes details such as changes in job descriptions and training existing employees to make the transition into healthcare’s “Age of Analytics.” As we have seen with EMRs, it is not sufficient to simply install the technology, but it must be Meaningfully Used.
HealthLeaders: ‘The goal was to populate an information repository to let service line workers go after whatever they need, Schooler says. “We’ve been heads down at this for maybe a year and three or four months, [and] we believe there is a three- to five-year initial deployment to get to where there is, what we would consider, a critical mass of information from across the organization.”’
  • Pattern #4:  The number of source systems whose content is represented in an enterprise data warehouse grows in proportion to the data literacy of the organization.
As the data literacy of the organization increases, so does the demand for new data content in the data warehouse. Again, we see metaphors to public libraries, where the demand for new and more challenging reading material increases as the surrounding community becomes more literate. In some healthcare organizations, where there is a high literacy rate and pent-up demand for data, the number of source systems feeding the data warehouse will grow exponentially at first, to as many as 50 source systems within five years.
CIOs must adopt data warehouse designs that can adapt to this demand for new source content and keep pace with demand, otherwise they will lose the faith and support of their analytic customers and, worse, will be blamed for suppressing the analytic maturity of the organization. Mapping new sources of data content into a data warehouse can be a very laborious and time-consuming process, especially for data warehouse designs that are Inmon-style, enterprise data models.
CIOs must anticipate this demand for new data content and plot their staffing strategy accordingly. The most valuable staff members in these source-system mapping projects are the source-system data stewards and application experts who are supporting production-system activities. They typically can’t be removed from their production system duties at a moment’s notice to focus for weeks at a time on the data-warehouse data-mapping project. The CIO can either hire this expertise for permanent assignment to the data warehouse or plan far enough ahead to ensure that the source system data stewards and application experts are available to the data warehouse team when needed. Again, timing is critical.
These are just a few of the many patterns and resultant principles that emerge in the analytic culture of an organization. Understanding and anticipating them is fundamentally important to the success of a data warehouse and analytic strategy. By observing their subtle appearance in the early-adopter organizations that are highlighted in this article, and looking for the same in their organizations, CIOs can plan and execute with even greater efficiency.

Monday, September 17, 2012

Lincoln, Leadership and the Power of Pure Motives

Today is the 150th anniversary of the Battle of Antietam where fellow countryman killed 23,000 Americans in less than two days.

Up to this point, Lincoln's decision-making motives in the war focused primarily on retaining the Union-- fighting secessionists-- not so much about the abolition of slavery.  He didn't want to be known as the President that let the Union slip away.  In fact, Lincoln was afraid to confront slavery head-on, for fear of losing the support of the North-South border states who were non-committal about the issue of slavery.  But in his diary, you can see the stress of losing the war, combined with the death of another son, Willie, 11, drive him into a more spiritual period of reflection.  He starts to question the validity of his Union-saving motive.  He realizes that the more fundamental and honorable motive is the abolishment of slavery-- that all men are created equal-- and that saving the Union and his legacy as President are relatively insignificant.  Antietam's blood bath precipitated his courage.  He decides to disregard his concern about the border states and henceforth, all of his decisions about fighting the war will be motivated by the elimination of slavery, with the retention of the Union a distant second concern.  He issues the Emancipation Proclamation immediately after Antietam.

Within a matter of weeks, the tide of the war turned in favor of the North.  Nothing else in the environment changed except Lincoln's motives and his courage to stand up in favor of the more fundamental and pure issue of freedom, even if it meant the loss of political support.  It was as if God were watching, withholding support, until Lincoln addressed the more pure motive.

For me, reading and learning about Lincoln in this context and this turn of events in the war was a watershed moment in my life and made me realize the power of pure motives-- not driven by ego, greed, irrational fear, careerism, or vengeance-- but rather by love, fundamental justice, charity, and compassion.  For many years leading up to this, first as a young military officer and then in civilian professional life, I searched for a universal framework for decision making that I knew would result in the best outcome...the proper outcome.  The Bible was the best framework I could find previously, but in diverse religious environments, leading diverse people, the Bible did not always provide the guidance I was searching for.  Likewise, for me at least, the Bible is sometimes complicated and contradictory-- compassionate and forgiving in the New Testament, harsh and unforgiving in the Old Testament.  When I witnessed in Lincoln's own writing and speeches, the slow spiritual emergence of Lincoln's change in motives and the impact those new motives had on the outcome of the Civil War,  it was the single most insightful moment of my life.  It launched me down a path in which I now constantly check and explore my motives, as best human fallibility will allow, in hopes that all personal and professional decisions and behaviors are firmly rooted in universal truths, even if the short term outcome looks bleak in doing so.  This manner of thinking changed everything about my life...everything.  All of my decisions improved in the near-term, and it relieved the long-term burden of retrospective regret, with the comforting knowledge that I did the best I could... my motives were as pure as possible... and that whatever followed was as destiny meant it to be.  Of course, I do not mean to imply that I abide by and practice this philosophy completely, only that I try, and over the journey of life, hope for the practice to occupy a larger and larger part of my life.  It's worth noting, that given the fallibility of the human ego, and the complexity of human relations, finding the right and pure motive is not always an easy task.  It might take days or weeks of background contemplation to finally arrive, but once you find it, you will know.  You will find a deep and calm sense of truth...and you will know.

When faced with a decision-- a choice-- no matter how small or insignificant-- consider your motives and adjust them towards fundamental purity.  The accumulation of small everyday choices guided in this fashion, seems to me, no less important than applying it in more singular and significant decisions.

Saturday, August 25, 2012

Neil Armstrong: "Houston, uh...Tranquility base here. The Eagle has landed."


Those were the first chilling words from Neil Armstrong. Later would come his more famous, "That's one small step for man; one giant leap for mankind."

As a young Air Force officer, I was lucky enough to meet Neil Armstrong. He gave a lecture to a group of junior officers from all four branches of the military about the role of "calm courage" in leadership. In essence, he said, courage is not the absence of fear, but rather the strength and determination to overcome it. Preparation, study and practice are the foundation for courage. "Fearlessness", he said, was "short ‘for foolish ego’". He said “My greatest achievement was not walking on the moon, but rather landing on the moon.” Then he told the story.  Here's the NASA video and audio recording of the 15-minute descent of Armstrong and Aldrin, landing the Eagle.

He took us through a minute-by-minute tour of this recording, describing the background and details of what was happening and what he was thinking about-- "There were all sorts of things going wrong, not according to plan, but the public never really knew anything about it, especially at the time." As best my memory can recall, here’s the story that he told:

At 7:45 you’ll hear the crew say, “Program Alarm” then “”1202” which was the specific alarm code. That’s the beginning of multiple alarms and confusion.

At 8:06, you’ll hear Armstrong impatiently insist, after no feedback from Mission Control, “Give us a reading on the 1202 program alarm.”

At 8:12, Mission Control, “Roger, we’re go on that alarm.” But, they weren’t 100% confident, as was revealed later.

I’m not sure when, but at some point in the above, Armstrong took over manual flight control. At 9:38, he tries to go back into auto control and it seems to hold.

At 11:38 you’ll hear Aldrin call out a “Program Alarm” and “1201”. That was the beginning of another series of similar alarms and confusion about the status of the flight control computer, notably the descent radar system. The crew and Mission Control had never seen these types of alarms in the simulator; they were scrambling to decide whether to abort the landing or not. As I recall him telling the story, the descent radar was placed in the wrong mode, which overloaded the flight control computer and caused the LEM to overshoot the primary landing zone by several miles and burn more fuel than planned. Neil Armstrong again took over manual flight control and was having a hard time finding a place to land, fearful that they would land on the edge of a crater or boulder and the LEM would tip over. So, imagine this… Armstrong manually flying, with Aldrin peering out the window for a clear spot and providing “eyeball” callouts of altitude and drift. 

At 14:20, you’ll hear someone in the background at Mission Control callout "60 seconds" and at 14:50, "30 seconds". That's Mission Control warning the crew that they had only 30 seconds of fuel left in the LEM propulsion system. Armstrong soon landed and they quickly went through their engine shut down procedures… with only 14 seconds of fuel remaining. They were supposed to land with at least one minute of fuel remaining, to ensure that they could safely return to the orbiting Command Module, Columbia. 

After landing and shutting down engines, Armstrong utters his famous words, "Houston, uh...Tranquility base here.  The Eagle has landed." Mission Control then says "Roger, Tranquility, we copy you on the ground. You gotta bunch of guys about to turn blue. We're breathing again. Thanks a lot." Then Armstrong replies, “Thank YOU” which always makes me chuckle to hear. In Armstrong’s thanks, you sense his acknowledgement of Mission Control’s role, plus his genuine Ohio, midwestern small town courtesy, as if he’s having dinner at someone’s house…not having just landed on the moon.

Apollo 11 was one of humanity's greatest achievements. It brought the world together at a time in the 1960s when the world seemed like it was coming apart. But Apollo 11 was also a hair-width away from one of humanity's greatest tragedies. Imagine how we would have felt if the Eagle had landed, but without enough fuel to takeoff and return to the Command Module. Two astronauts would have died a very slow and gruesome death, and we would have never looked up at the moon the same way, again. Neil Armstrong's calm, cool, courage, along with Aldrin’s teamwork, and calm heads in Mission Control, were the difference in that hair-width boundary.

For whatever reason, the impact of this story didn't hit me until later, when I worked for TRW. I was lucky enough to sit in on another related lecture, in honor of the Apollo 11 25th anniversary, and listen to some of the TRW engineers who were in Mission Control for Apollo 11. They designed and built the descent engines for the Lunar Module ("LEM"). They knew the untold story of the near disaster that I heard from Armstrong, years before. 

Armstrong was not comfortable with the attention he garnered from what is still the greatest adventure of humankind.  He could have been the total opposite...the most arrogant man in the world, yet he was humbled.  I wish we had more corporate and political leadership who were uncomfortable in those roles-- the reluctant leaders, pushed into their role, not pushing others out of the way to get there.

We had our share of trouble in the 60s and 70s, when Apollo brought the world together.  Even with those problems, I think there was always an underlying sense of confidence in ourselves—in America-- and from the rest of the world that the US could eventually overcome any challenge. We need to get that attitude back.  We need to earn it back.

Update to this blog: A dear friend, Susan Pollack, found a detailed write-up about this story that you can read here: Apollo 11's Scariest Moments: Perils of the 1st Manned Moon Landing.

Monday, August 6, 2012

Hacking a Nuclear Conferencing System


On my LinkedIn profile, there’s a blurb in my bio that generates some fun conversations.  That blurb is:  “In 1988, he hacked into the national nuclear warning system from a public pay phone.”  At the request of several friends and colleagues, here’s the story behind that blurb…

From 1983-1991, I was an information systems officer in the US Air Force, the last two years of that in the reserves.  My entire career was centered on nuclear warfare operations, supporting what was then known as the Strategic Air Command (SAC).  The pinnacle of that job was serving as a member of the battle staff on an aircraft known at the Looking Glass.  There were 14 of us who functioned as airborne CIOs, primarily managing the battle staff information systems on the plane and the vast array of information systems that were ground and space-based, associated with nuclear warfare.  As an Airborne Launch Control Officer, the CIO was also responsible for “turning keys” with the Operations Controller to launch all 1,000 nuclear ballistic missiles in the US inventory.  Lynda Sabin, to this day a dear friend, was also one of those 14 officers and now she, too, is in healthcare analytics.  Interesting coincidence.  The Wikipedia write-up is available here, but in short, the Looking Glass aircraft and battle staff were responsible for assuming command of all US nuclear forces in the event of war, and managing that war from the safety and survivability of the air-- in a nuclear war, not much would be left of ground-based command centers like the Pentagon.  The Looking Glass was the anchor in a larger “World Wide Airborne Command Post” organization that consisted of numerous aircraft operating from bases around the world that were connected by a global wireless network.

That global wireless network between the orbiting aircraft was also connected to wireless ground entry points that were managed by AT&T.  These ground entry points took our wireless signal for voice and data traffic and routed it via landlines to SAC and other military forces, as well as the White House.  There were two very critical voice conferencing systems that operated on isolated AT&T ground networks:  the SAC Operational Conferencing System (SOCS, pronounced sox) and the Joint Chiefs of Staff Alerting Network (JCSAN, pronounced jay-san).  SOCS was the voice conferencing system used primarily to manage SAC nuclear forces in the event of an emergency—i.e., Air Force bombers, cruise missiles, and intercontinental ballistic missiles.  JCSAN was the voice conferencing system used to manage nuclear and other military operations from the Pentagon, including the nuclear forces associated with the Army and Navy, but Air Force, too.  AT&T provided the circuits and technical support for each of these physically isolated and independent networks.

As those who have served will attest, life in the military can be summarized as hours and hours of pure boredom interrupted by moments of shear terror.  During those hours of boredom, I would read and study everything I could find—the latest intelligence reports, force status reports; military plans, especially the President’s “Black Book” and the Single Integrated Operational Plan, or SIOP, which were the massive handbooks for nuclear warfare and reconstitution of US society after a nuclear exchange.  I would also study the configuration of the information systems that we were tasked with managing, and along with our IS teams, we would quiz each other on the trivia and nuances of those configurations. 

SOCS and JCSAN were supposed to be physically separate networks from AT&Ts public telephone switching systems.  All three—the public telephone network, SOCS and JCSAN were supposed to have an “air gap” between them.  Under certain emergency situations, the air gap could be manually closed, like patching two servers together or bypassing a firewall with a hardwire connection.   The air gap and physical separation were a very deliberate design.  SAC had its need for a conferencing system to manage SAC specific-issues and the Pentagon had its need for managing more global issues across all of the military branches by using JCSAN. In the worst case communications scenario, if the President or National Command Authority needed to communicate to military forces from a public telephone, the public telephone network could be manually connected to JCSAN and SOCS.  Like a separate highway, AT&T’s publicly switched telephone networks were supposed to be completely separate from both the SOCS and JCSAN, with the exception of a drawbridge that could be lowered in an emergency to allow traffic to travel among the three.  But, over a period of several months studying and quizzing members of my IS team, I saw what I thought was an oddity in the design of the AT&T networks that could allow public access to the SOCS and JCSAN networks-- a backdoor that was open… a drawbridge that was already lowered.

For several of those boring nights aboard the Looking Glass, I studied the JCSAN, SOCS, and private telephone network configurations and occasionally talked to AT&T engineers in the ground entry points, asking them questions that focused on the oddity that caught my attention-- I see a way to connect AT&T’s public telephone system into the JCSAN, without manual intervention—without the patch cord across the air gap that separated them.  No, that’s not possible.  Naturally, when you start to see something like I was seeing—when the pattern in the puzzle first emerges-- you can’t believe what you’re seeing.  You keep telling yourself that you misunderstand something… surely it can’t be so. And that’s what I did for weeks… I wrestled in disbelief.

I mentioned what I thought I saw—this backdoor-- to a few colleagues and teammates, but none of them really took it seriously because even I didn’t believe it completely.  But the more I thought about it, the more I looked at the puzzle, the more I was convinced that the backdoor existed.  My mind wouldn’t let it go… and there was only one way to confirm it or prove that I was wrong—try to hack the backdoor.  If I were right and the door opened, it would mean uncovering a big stinky mess associated with one of the most important information systems assets in the US military.  The JCSAN was connected to the “red phone” in the Oval Office and Situation Room.  If I were wrong and the hack failed, it would go unnoticed—a failed hack of no significance.  But that’s not something you “just try” in the military as a young officer.  If you uncover that big stinky mess, it’s going to splatter on a lot of people’s toes, all the way up the ladder.  But if I didn’t uncover it and reveal the problem, what if someone else did?  What if one of the “bad guys” found it and kept it a secret until there was a national emergency… and then they exploited it and disrupted our ability to manage US nuclear forces?  This was in 1987 and ’88, when the Cold War was on steady simmer and the stability of the Soviet Union was just beginning to unravel.  We weren’t sure who would be in charge of Soviet nuclear weapons nor what their state of leadership mind might be.  And I was only 28 years old at the time… it was a stressful burden and decision at a young age, but in a twisted way, the burden was captivating.

One night, at the conclusion of a Looking Glass flight, a group of us from the battle staff went to a local watering hole—Gilmore Lake Tavern—near Bellevue, Nebraska, not far outside the gates of SAC Headquarters in Omaha.  During idle conversation over beer, I mentioned finding what appeared to be a backdoor to the JCSAN, through the public telephone network.  To some of the teammates at Gilmore’s Tavern that night, it was old news—I had told them before—and they lightheartedly dismissed it.  To other teammates, it was just another crazy scheme from The Beave (they nicknamed me The Beaver, as in Ozzie and Harriet’s son, because of the 1950’s flattop haircut that made me look more like 18 than 28 years).  Major Chuck Horton, a dear friend, laughed and bet me a pitcher of beer that I was wrong.  No way was there a backdoor to the JCSAN.  It’s time to put The Beave’s rumor to rest.  Put up or shut up Sanders.  Open the door or quit talking about it.

I didn’t respond for a few minutes, percolating on whether it was worth calling his bet or not.  A pitcher of beer was meaningless; the challenge of the bet was about the risk of adventure.  While I percolated, so did the beer that I’d been drinking and the liquid courage soon overtook the inhibitions.

The dialing sequence to the backdoor was in the breast pocket of my flight suit.  I’d been carrying it around with me for about a month.  I pulled it out, looked at Horton and said, “OK, here we go.  Let’s see what happens.”  He laughed and stayed in his chair.  No one else at the table was paying much attention.  I walked across the bar to the pay phone, put in a quarter, and started dialing the sequence, traversing my way across AT&T’s publicly switched network.  Click, click, click… I would enter a public network switch, get a dial tone, and dial into another.  As I recall, I traversed about 12 switches and networks.

And then I reached what I predicted was the backdoor—the switch that bridged the public telephone system with JCSAN.  I paused like I was standing in front of a real door, stared at it, took a breath, and dialed the last sequence.  If the sequence was wrong, I would get another dial tone into an unknown switch, like opening a door into a an empty room; or I would get a trunk busy signal indicating a dead end…the dead end air gap between the public system and the JCSAN.  If the door opened, I would set off a worldwide alert, known as a warble tone, in all of the US military’s major command centers, including the Pentagon.  The battle staffs of those command centers would pick up their “red phone”, join the conference call, and prepare to receive emergency action messages (EAMs) or other instructions from the Joint Chiefs of Staff or National Command Authorities.

A warble tone went off.  The backdoor was open.  The command centers started answering—This is NORAD. This is NMCC.  This is CENTCOM. This is Space Command…and so on.  

I wasn’t prepared.  I hadn’t actually thought completely through the scenario—“What if the door opens?”  I sat there with the black handset from the Gilmore Lake Tavern phone in my hand, pressed to my ear, stunned for a few seconds, speechless. I looked across the bar at Horton and mouthed the words, “Holy &#*%!”, my eyes wide open. He leaned forward in his chair with a frown of concern.  Then I silently mouthed a second time, “Holy &#*%!”.  Horton’s jaw dropped and his eyes popped. 

A waitress walked by, glanced and smiled at me and then Horton, wondering about the exchange of all the odd facial gestures.  She looked back at me quizzically and paused just long enough to become a party to the event that was unfolding.  I reacted instinctively, like a 12-year old caught red handed in a serious prank—and handed the phone to her.

She looked at me, “What…?  Who is it?  What do you want me to do?”

I said the first thing that popped in my head. 

“Say ‘Hello, General’.” 

“Say ‘Hello, General’?” 

“Yes, say ‘Hello General’.” 

And that’s what she said, “Hello General”.  Her voice came across loud and clear to every US major command center and nuclear battle staff in the world.   I grabbed the phone back, now in a panic, and stuck it to my ear.  My mind racing with oh my god, what have I done??  There was total, stunned silence on the other end.  After a few seconds, someone finally asked, “Who is this?  Who convened this conference?”

Slam.  I hung up the phone and stood there staring at it.  Sanders, you’ve gone a little too far this time, was the voice inside my head.  The waitress asked me, “Who was that?” but I couldn’t respond.  I looked at Horton and shook my head, side to side.  I walked across the bar, back to the table, slowly, eyes wide, mouth tightly closed. 

“It worked”, I told Horton. 

“No way”, he said without excitement, only stern denial. 

“Oh yes—it worked.” 

I told him what happened.  He responded,  “You can’t tell anyone you did this.  Don’t say a #$%&* word about this to anyone!” 

It was a sleepless night, excited like an explorer who stumbled on an important but forbidden discovery.  I wanted to talk about it, but knew quiet discretion was the wiser choice.

The next day, “the incident” was the topic of discussion all over our squadron, and apparently SAC headquarters, too.… and probably all over the military command centers and the Pentagon, as well.  A few of my SAC and Looking Glass teammates knew that I had been talking about this backdoor for quite some time.  They knew it was most likely me behind the incident—they would glance at me in a knowing way-- but they didn’t say a word.  A formal investigation was launched.  I wasn’t involved in the investigation, nor was I quizzed or implicated.  To this day, I don’t know what became of the situation.  I heard that AT&T was summoned to explain, but I don’t really know.   I purposely laid low, no interest in attracting attention to myself, but rather satisfied in the confidence that the backdoor would be closed.

I never had any interest in testing the backdoor again, assuming that it had been closed, but also feeling like I’d used one of my nine lives the first time-- no need to risk another. Over time, as the seriousness of the incident subsided and the military became more concerned about other matters, such as the demise of the Soviet Union, the incident became more openly discussed, often times with a sense of humor about a significant but ultimately harmless caper.  I didn’t openly discuss and fully admit to being the culprit until a couple of years after I resigned from the Air Force and was working for TRW where the incident became a small badge of honor among a cult of similarly mischievous “kids”.  Later, our team at TRW, led by Ron Gault, was hired by the National Security Agency to conduct counter espionage and threat analysis on the entire US nuclear command and control system; those hours of boring study on the Looking Glass paid off nicely under the NSA project.

There are pros and cons to an obsessively curious personality.  I’ve had an interesting life because of that curiosity, but it can push you to the borders of trouble, too.  In this case, it was more than curiosity that pushed me into the incident.  I couldn’t stand not knowing if that backdoor would open or not, and my somewhat impetuous side pushed me to indulge that curiosity in a very unusual setting—Gilmore Lake Tavern.  It would have been just as easy to test the backdoor in a controlled and official Air Force setting, which crossed my mind.  But, the sense of adventure in opening that door by myself, without any supervision, was far more appealing.  The color in life comes from unusual events and settings.  As the janitor for the Liberty Bell museum in Philadelphia told me one time, “If it weren’t for the crack, it would just be a bell and it wouldn’t mean a thing to nobody.”

Sometimes, if life doesn’t do it for you, you have to put a crack in the bell.  J

Thursday, July 19, 2012

The Emperor's New Data Model

That was the title of the post to a LinkedIn group ("Healthcare Data Warehousing Association") by my friend and colleague, Dr. Jonathan Einbinder, of Partners Healthcare.  Here's a copy and paste of his post and my response.

Dr. Einbinder:  "In real-world terms, how is a comprehensive healthcare data model (e.g. like the ones provided by Oracle or IBM), helpful in moving forward an analytics agenda? Seems to me that potential value lies in providing a standard target for ETL and in building (or acquiring) content/reports on top of the model. That is the idea, anyway. In pragmatic terms, I wonder if it makes more sense to focus on standardizing a few key dimensions... (anyone remember IHC's Common Data Bus). I would like to hear how other provider organizations approach this."

Dale Sanders: The comprehensive enterprise data model sounds great, in theory, but it fails miserably in practice. I've lived it first-hand and watched it second-hand numerous times now. Of course you know my bias for the Intermountain (IHC) bus architecture, but that bias comes from a good place...pragmatic common sense. 

The origins of the bus approach go back much further than Intermountain (IHC), actually, back to an Air Force project that I worked on in the early 1990s. The bus architecture emerged after the failure of a $60M data warehouse project to pull 450 source systems into a common analytic model for nuclear weapon systems management. It took three years and 200 consultants to develop the enterprise model and ETL for that project. The ETL was so complicated, we spent all of our time maintaining and fixing the problems. When a new source system was introduced, the ETL scripts took months to write; there was no agility. When we exposed the enterprise data model to the analysts, they stared blankly because they had lost all familiarity with the content of their source systems, even though we had a very robust metadata repository, showing all the mappings-- the analysts simply didn't recognize the "enterprise" representation of their data. Finally, the number of analytic use cases that required an enterprise model was less than 2%. We tracked, measured and surveyed the need, post facto.

An electrical-computer engineer, Scott Birbeck, who was a friend and colleague, but not assigned to this particular project, was watching this disaster from a distance. One day he said to me, "Why don't you guys stop trying to make all the data look the same and just make it look the same only where you have to, like we (EE's) do when we build a bus for a computer? We could care less about the details of the devices on the bus, we just care about how they connect to the bus so they can communicate to everything else. Isn't this basically the same problem?" We went to the white board and started sketching.

The next day, a few of us who were desperately trying to pull this data warehouse disaster from the fire, sat down and defined the format and naming conventions for the "core data elements" required to communicate across the "data bus". It took less than a full day to do that for 450+ source systems. The next day, two of us started adding attributes to the source system data models in the staging area, to reflect these core data elements (previously, we had not allowed analysts to access their data in the staging area, which was another mistake in retrospect). In about a week, we had the ability to query across a large number of the most important source systems using these new foreign keys in the data bus.

Going against the policies of the project manager, we secretly gave analysts direct access to the source system data in the staging area and told them how to take advantage of the new data bus...and away they went. They loved it. Soon, the analysts started asking for extracts of the data from the source systems-- data mart subsets of the data that pertained to their particular analytic need; so we built those, too-- they didn't need nor want to constantly pull data from an enterprise model.

We purposely kept this all very secret until there was enough momentum and support from the analysts behind the new approach that the success could not be denied. We didn't want to take this secretive route, but had no choice because the project manager was dogmatically, egocentrically tied to the enterprise model strategy. He soon left the company and still despises those of us who were involved in the recovery, but the Air Force Colonel who was the executive sponsor loved us, as did the analysts.

In three months, the project went from disaster to heroic. The design is still operating and evolving nicely, now almost 20 years later. Oracle told us that it was their largest data warehouse in the world at the time.

SpaceX Inspirations

SpaceX launched a two-astronaut crew yesterday, on a mission to dock with the International Space Station. It was the first human spaceflig...