Saturday, August 25, 2012

Neil Armstrong: "Houston, uh...Tranquility base here. The Eagle has landed."

Those were the first chilling words from Neil Armstrong.  Later would come his more famous, "That's one small step for man; one giant leap for mankind."

As a young Air Force officer, I was lucky enough to meet Neil Armstrong. He gave a speech to a group of junior officers from all four branches of the military about the role of "calm courage" in leadership. In essence, he said, courage is not the absence of fear, but rather the strength and determination to overcome it. Preparation, study and practice are the foundation for courage . "Fearlessness", he said, was "short for foolish ego". He said that his greatest achievement was not walking on the moon, but rather landing on the moon. Here's the NASA video and audio recording of the 15-minute descent of Armstrong and Aldrin, landing the Eagle.

He took us through a minute-by-minute tour of this recording, describing the background and details of what was happening and what he was thinking about-- "There were all sorts of things going wrong, not according to plan, but the public never really knew anything about it, especially at the time." As you listen to the last couple of minutes of this recording, you hear someone in the background say, "60 seconds" and later "30 seconds". That's mission control warning the crew that they had only 30 seconds of fuel left in the LEM descent propulsion system. As I recall him telling the story, the descent radar was placed in the wrong mode, which overloaded the flight control computer and caused the LEM to overshoot the primary landing zone by several miles and burn more fuel than planned. Neil Armstrong took over manually and was having a hard time finding a place to land, fearful that they would land on the edge of a crater or boulder and the LEM would tip over. Armstrong landed and shut down engines with 14 seconds of fuel remaining. They were supposed to land with 2 minutes of fuel remaining, to ensure that they could safely return to the orbiting Command Module, Columbia.  After landing and shutting down engines, Armstrong says his famous words, "Houston, uh...Tranquility base here.  The Eagle has landed." Mission Control then says "You have a bunch of guys about to turn blue. We're breathing again. Thanks a lot."

Apollo 11 was one of humanity's greatest achievements. It brought the world together at a time in the 1960s when the world seemed like it was coming apart. But Apollo 11 was also a hair-width away from one of humanity's greatest tragedies. Imagine how we would have felt if the Eagle had landed, but without enough fuel to takeoff and return to the Command Module. Two astronauts would have died a very slow and gruesome death, stranded on the moon. Neil Armstrong's calm, cool, courage, and thinking was the difference in that hair-width boundary.

For whatever reason, the impact of this story didn't hit me until later, when I worked for TRW. I was lucky enough to sit in on another related lecture, this one by some of the old TRW engineers who were in Mission Control for Apollo 11, in honor of the 25th anniversary. They designed and built the descent engines for the Lunar Module ("LEM"). In their lecture, they spent more time emphasizing the significance of the disaster that Apollo 11 avoided, and how cool and level headed Armstrong was during the intense pressure to land.

Armstrong was not comfortable with the attention he garnered from what is still the greatest adventure of humankind.  He could have been the total opposite...the most arrogant man in the world.  I wish we had more corporate and political leadership that were uncomfortable in those roles-- the reluctant leaders, pushed into the role, not pushing others out of the way to get there.

We had our share of trouble in the 60s and 70s, when Apollo brought the world together.  Even with those problems, I think there was always an underlying sense of confidence in ourselves and from the rest of the world that the US could eventually overcome any challenge. We need to get that attitude back.  We need to earn it back.

Update to this blog: A dear friend of mine, Susan Pollack, found a detailed write-up about this story that you can read here: Apollo 11's Scariest Moments: Perils of the 1st Manned Moon Landing.

Monday, August 6, 2012

Hacking a Nuclear Conferencing System

On my LinkedIn profile, there’s a blurb in my bio that generates some fun conversations.  That blurb is:  “In 1988, he hacked into the national nuclear warning system from a public pay phone.”  At the request of several friends and colleagues, here’s the story behind that blurb…

From 1983-1991, I was an information systems officer in the US Air Force, the last two years of that in the reserves.  My entire career was centered on nuclear warfare operations, supporting what was then known as the Strategic Air Command (SAC).  The pinnacle of that job was serving as a member of the battle staff on an aircraft known at the Looking Glass.  There were 14 of us who functioned as airborne CIOs, primarily managing the battle staff information systems on the plane and the vast array of information systems that were ground and space-based, associated with nuclear warfare.  As an Airborne Launch Control Officer, the CIO was also responsible for “turning keys” with the Operations Controller to launch all 1,000 nuclear ballistic missiles in the US inventory.  Lynda Sabin, to this day a dear friend, was also one of those 14 officers and now she, too, is in healthcare analytics.  Interesting coincidence.  The Wikipedia write-up is available here, but in short, the Looking Glass aircraft and battle staff were responsible for assuming command of all US nuclear forces in the event of war, and managing that war from the safety and survivability of the air-- in a nuclear war, not much would be left of ground-based command centers like the Pentagon.  The Looking Glass was the anchor in a larger “World Wide Airborne Command Post” organization that consisted of numerous aircraft operating from bases around the world that were connected by a global wireless network.

That global wireless network between the orbiting aircraft was also connected to wireless ground entry points that were managed by AT&T.  These ground entry points took our wireless signal for voice and data traffic and routed it via landlines to SAC and other military forces, as well as the White House.  There were two very critical voice conferencing systems that operated on isolated AT&T ground networks:  the SAC Operational Conferencing System (SOCS, pronounced sox) and the Joint Chiefs of Staff Alerting Network (JCSAN, pronounced jay-san).  SOCS was the voice conferencing system used primarily to manage SAC nuclear forces in the event of an emergency—i.e., Air Force bombers, cruise missiles, and intercontinental ballistic missiles.  JCSAN was the voice conferencing system used to manage nuclear and other military operations from the Pentagon, including the nuclear forces associated with the Army and Navy, but Air Force, too.  AT&T provided the circuits and technical support for each of these physically isolated and independent networks.

As those who have served will attest, life in the military can be summarized as hours and hours of pure boredom interrupted by moments of shear terror.  During those hours of boredom, I would read and study everything I could find—the latest intelligence reports, force status reports; military plans, especially the President’s “Black Book” and the Single Integrated Operational Plan, or SIOP, which were the massive handbooks for nuclear warfare and reconstitution of US society after a nuclear exchange.  I would also study the configuration of the information systems that we were tasked with managing, and along with our IS teams, we would quiz each other on the trivia and nuances of those configurations. 

SOCS and JCSAN were supposed to be physically separate networks from AT&Ts public telephone switching systems.  All three—the public telephone network, SOCS and JCSAN were supposed to have an “air gap” between them.  Under certain emergency situations, the air gap could be manually closed, like patching two servers together or bypassing a firewall with a hardwire connection.   The air gap and physical separation were a very deliberate design.  SAC had its need for a conferencing system to manage SAC specific-issues and the Pentagon had its need for managing more global issues across all of the military branches by using JCSAN. In the worst case communications scenario, if the President or National Command Authority needed to communicate to military forces from a public telephone, the public telephone network could be manually connected to JCSAN and SOCS.  Like a separate highway, AT&T’s publicly switched telephone networks were supposed to be completely separate from both the SOCS and JCSAN, with the exception of a drawbridge that could be lowered in an emergency to allow traffic to travel among the three.  But, over a period of several months studying and quizzing members of my IS team, I saw what I thought was an oddity in the design of the AT&T networks that could allow public access to the SOCS and JCSAN networks-- a backdoor that was open… a drawbridge that was already lowered.

For several of those boring nights aboard the Looking Glass, I studied the JCSAN, SOCS, and private telephone network configurations and occasionally talked to AT&T engineers in the ground entry points, asking them questions that focused on the oddity that caught my attention-- I see a way to connect AT&T’s public telephone system into the JCSAN, without manual intervention—without the patch cord across the air gap that separated them.  No, that’s not possible.  Naturally, when you start to see something like I was seeing—when the pattern in the puzzle first emerges-- you can’t believe what you’re seeing.  You keep telling yourself that you misunderstand something… surely it can’t be so. And that’s what I did for weeks… I wrestled in disbelief.

I mentioned what I thought I saw—this backdoor-- to a few colleagues and teammates, but none of them really took it seriously because even I didn’t believe it completely.  But the more I thought about it, the more I looked at the puzzle, the more I was convinced that the backdoor existed.  My mind wouldn’t let it go… and there was only one way to confirm it or prove that I was wrong—try to hack the backdoor.  If I were right and the door opened, it would mean uncovering a big stinky mess associated with one of the most important information systems assets in the US military.  The JCSAN was connected to the “red phone” in the Oval Office and Situation Room.  If I were wrong and the hack failed, it would go unnoticed—a failed hack of no significance.  But that’s not something you “just try” in the military as a young officer.  If you uncover that big stinky mess, it’s going to splatter on a lot of people’s toes, all the way up the ladder.  But if I didn’t uncover it and reveal the problem, what if someone else did?  What if one of the “bad guys” found it and kept it a secret until there was a national emergency… and then they exploited it and disrupted our ability to manage US nuclear forces?  This was in 1987 and ’88, when the Cold War was on steady simmer and the stability of the Soviet Union was just beginning to unravel.  We weren’t sure who would be in charge of Soviet nuclear weapons nor what their state of leadership mind might be.  And I was only 28 years old at the time… it was a stressful burden and decision at a young age, but in a twisted way, the burden was captivating.

One night, at the conclusion of a Looking Glass flight, a group of us from the battle staff went to a local watering hole—Gilmore Lake Tavern—near Bellevue, Nebraska, not far outside the gates of SAC Headquarters in Omaha.  During idle conversation over beer, I mentioned finding what appeared to be a backdoor to the JCSAN, through the public telephone network.  To some of the teammates at Gilmore’s Tavern that night, it was old news—I had told them before—and they lightheartedly dismissed it.  To other teammates, it was just another crazy scheme from The Beave (they nicknamed me The Beaver, as in Ozzie and Harriet’s son, because of the 1950’s flattop haircut that made me look more like 18 than 28 years).  Major Chuck Horton, a dear friend, laughed and bet me a pitcher of beer that I was wrong.  No way was there a backdoor to the JCSAN.  It’s time to put The Beave’s rumor to rest.  Put up or shut up Sanders.  Open the door or quit talking about it.

I didn’t respond for a few minutes, percolating on whether it was worth calling his bet or not.  A pitcher of beer was meaningless; the challenge of the bet was about the risk of adventure.  While I percolated, so did the beer that I’d been drinking and the liquid courage soon overtook the inhibitions.

The dialing sequence to the backdoor was in the breast pocket of my flight suit.  I’d been carrying it around with me for about a month.  I pulled it out, looked at Horton and said, “OK, here we go.  Let’s see what happens.”  He laughed and stayed in his chair.  No one else at the table was paying much attention.  I walked across the bar to the pay phone, put in a quarter, and started dialing the sequence, traversing my way across AT&T’s publicly switched network.  Click, click, click… I would enter a public network switch, get a dial tone, and dial into another.  As I recall, I traversed about 12 switches and networks.

And then I reached what I predicted was the backdoor—the switch that bridged the public telephone system with JCSAN.  I paused like I was standing in front of a real door, stared at it, took a breath, and dialed the last sequence.  If the sequence was wrong, I would get another dial tone into an unknown switch, like opening a door into a an empty room; or I would get a trunk busy signal indicating a dead end…the dead end air gap between the public system and the JCSAN.  If the door opened, I would set off a worldwide alert, known as a warble tone, in all of the US military’s major command centers, including the Pentagon.  The battle staffs of those command centers would pick up their “red phone”, join the conference call, and prepare to receive emergency action messages (EAMs) or other instructions from the Joint Chiefs of Staff or National Command Authorities.

A warble tone went off.  The backdoor was open.  The command centers started answering—This is NORAD. This is NMCC.  This is CENTCOM. This is Space Command…and so on.  

I wasn’t prepared.  I hadn’t actually thought completely through the scenario—“What if the door opens?”  I sat there with the black handset from the Gilmore Lake Tavern phone in my hand, pressed to my ear, stunned for a few seconds, speechless. I looked across the bar at Horton and mouthed the words, “Holy &#*%!”, my eyes wide open. He leaned forward in his chair with a frown of concern.  Then I silently mouthed a second time, “Holy &#*%!”.  Horton’s jaw dropped and his eyes popped. 

A waitress walked by, glanced and smiled at me and then Horton, wondering about the exchange of all the odd facial gestures.  She looked back at me quizzically and paused just long enough to become a party to the event that was unfolding.  I reacted instinctively, like a 12-year old caught red handed in a serious prank—and handed the phone to her.

She looked at me, “What…?  Who is it?  What do you want me to do?”

I said the first thing that popped in my head. 

“Say ‘Hello, General’.” 

“Say ‘Hello, General’?” 

“Yes, say ‘Hello General’.” 

And that’s what she said, “Hello General”.  Her voice came across loud and clear to every US major command center and nuclear battle staff in the world.   I grabbed the phone back, now in a panic, and stuck it to my ear.  My mind racing with oh my god, what have I done??  There was total, stunned silence on the other end.  After a few seconds, someone finally asked, “Who is this?  Who convened this conference?”

Slam.  I hung up the phone and stood there staring at it.  Sanders, you’ve gone a little too far this time, was the voice inside my head.  The waitress asked me, “Who was that?” but I couldn’t respond.  I looked at Horton and shook my head, side to side.  I walked across the bar, back to the table, slowly, eyes wide, mouth tightly closed. 

“It worked”, I told Horton. 

“No way”, he said without excitement, only stern denial. 

“Oh yes—it worked.” 

I told him what happened.  He responded,  “You can’t tell anyone you did this.  Don’t say a #$%&* word about this to anyone!” 

It was a sleepless night, excited like an explorer who stumbled on an important but forbidden discovery.  I wanted to talk about it, but knew quiet discretion was the wiser choice.

The next day, “the incident” was the topic of discussion all over our squadron, and apparently SAC headquarters, too.… and probably all over the military command centers and the Pentagon, as well.  A few of my SAC and Looking Glass teammates knew that I had been talking about this backdoor for quite some time.  They knew it was most likely me behind the incident—they would glance at me in a knowing way-- but they didn’t say a word.  A formal investigation was launched.  I wasn’t involved in the investigation, nor was I quizzed or implicated.  To this day, I don’t know what became of the situation.  I heard that AT&T was summoned to explain, but I don’t really know.   I purposely laid low, no interest in attracting attention to myself, but rather satisfied in the confidence that the backdoor would be closed.

I never had any interest in testing the backdoor again, assuming that it had been closed, but also feeling like I’d used one of my nine lives the first time-- no need to risk another. Over time, as the seriousness of the incident subsided and the military became more concerned about other matters, such as the demise of the Soviet Union, the incident became more openly discussed, often times with a sense of humor about a significant but ultimately harmless caper.  I didn’t openly discuss and fully admit to being the culprit until a couple of years after I resigned from the Air Force and was working for TRW where the incident became a small badge of honor among a cult of similarly mischievous “kids”.  Later, our team at TRW, led by Ron Gault, was hired by the National Security Agency to conduct counter espionage and threat analysis on the entire US nuclear command and control system; those hours of boring study on the Looking Glass paid off nicely under the NSA project.

There are pros and cons to an obsessively curious personality.  I’ve had an interesting life because of that curiosity, but it can push you to the borders of trouble, too.  In this case, it was more than curiosity that pushed me into the incident.  I couldn’t stand not knowing if that backdoor would open or not, and my somewhat impetuous side pushed me to indulge that curiosity in a very unusual setting—Gilmore Lake Tavern.  It would have been just as easy to test the backdoor in a controlled and official Air Force setting, which crossed my mind.  But, the sense of adventure in opening that door by myself, without any supervision, was far more appealing.  The color in life comes from unusual events and settings.  As the janitor for the Liberty Bell museum in Philadelphia told me one time, “If it weren’t for the crack, it would just be a bell and it wouldn’t mean a thing to nobody.”

Sometimes, if life doesn’t do it for you, you have to put a crack in the bell.  J

Nuclear and Healthcare Decision Making

Nuclear warfare operations was my data-driven decision making environment before the healthcare phase of my career. It was all about recogni...