So, I am working at this dotcom in New York City. They have some offices on the seventh floor of the building and more offices on the 19th floor -- and a really slow elevator between them. The actual servers, database machine, etc all lived in a datacenter a few miles away and we had a single T1 from the offices to the datacenter (for managment).
The DBA folks have some huge maintenance planned -- I'm not a database guy, but it sounded all impressive and they had been planning it for weeks. They had this big plan that involved failing over from one database server to the other, making some changes and then failing back -- there would be an outage, but it would be less than 10 seconds and the backends would just queue requests for that period.
Anyway, they start the maintenance and get to the critical 10 second bit... and suddenly their SSH sessions stops responding. After the initial panic that somehow unmounting the database had made the machine die we figured out that the network from the office to the database was down. I run of an log into the router -- the the serial to the T1 (that had been up and stable for more than a year) is now showing down/down. I hit our website (Internet access goes though a different, larger circuit) and the site is not showing "We're sorry, $COMPANY is down for maintenance" -- the backends have decided that they cannot reach the database for too long and are failing helthchecks -- not good.
I start calling the carrier and shout to the guy I work with "Joe, run up to the demarc on the 19th floor and check the light on the SmartJack" -- Joe scurries off and runs up 12 flights of stairs.. A few minutes later he rushes into the room with sweat dripping of his forehead. "What did hte lights show?!", I ask. Apparently he has forgotten the combination to the push button lock on the door -- I shout out "1-3, 2 5" and Joe runs off again -- up and down 12 flights of stairs.. He comes panting in and says the combination doesn't work. I swear quietly and explain that you need to press 1 and 3 together and then 2 and then 5. Joe runs off again. A few minutes later and the circuit is still down... Joe comes stumbling back in looking like he is about to pass out but also looking oddly sheepish.
"What happened? What did the lights show?"
"Well, I run up all the stairs..."
"Yes?!"
"And then type in number on the door...."
"Yes -- what did the lights show?!" (I am starting to lose my cool about now)
"And I find the demarc and I find the shelf and I look at the lights..."
"JOE! What did the lights show?!!!!"
"and then I remember -- I'm color-blind... I can't tell if the lights are red or green".