In 2006, with a strong case of senior-itis after putting in notice at my last employer, I was asked to document a number of things I did regularly or special things that may be done incorrectly that would be helpful to know. One of the documents I can share, because it pertains entirely to nonexistent servers, non existing servers, and a building prior to the full dissolution of my previous employer's detroit office.
It's among my favorites, one I actually saved a copy of because I liked the sense of humor I had about what had (more than) once been a very frustrating task for me to perform.
“What’s that burning smell?”
A guide on how to recover the Detroit office from a full, extended power outage.
Paul Timmins
Okay, if you’re reading this document, you’re probably pretty screwed. Here’s a step by step guide.
Breaking into the office: (This is only necessary if the door controllers are not working right) [editor's note: they never work right]
- The holdmagnetsare rated for 300 pounds of lateral force. Don’t waste yourtimedinking with trying to pull them open. You’ll break the door first. You have two options:
- 1⁄4” – 1⁄2” steel rod, 1 meter long. (you can get this at home depot for a buck, it might be threaded, but who cares) Slip this through the crack between the door and the jamb, and use it to push the emergency exit button. It requires about 2 pounds of force, so a meterstick might not be sufficient.
- MacGuyver method. Heat up a party balloon under your arm until it’s about 85-90 degrees. Stick its mass through the door, and hold it by its neck. Inflate it with a drinking straw. Release the balloon, such that it flies in the path of the motion sensor. This may take several tries.
- The fire panel can override the perimeter locks. There are two ways to trigger this. One is to call property management and convince them to override your locks at the fire panel. Option two is left as an exercise to the reader.
- So you’re in. Now what? Well, that server room lock is failshut. You can’t operate it in a power outage, and there’s no fire panel override. It’s a 6 pin tumbler lock with no special warding. A locksmith (or me!) can open it in under 15 minutes with a simple set of lockpicks. Prop this door because it WILL relock.
Getting everything happy:
- Ensure you have power to the entire office. Getting halfway through this to find out that you only have power to half the room is not good.
- Air conditioning is critical. The room should be below 70F. If it is above 85F, do not continue until the AC has cooled the room. This can occur if the systems run off UPS for a while, because the AC is not on the UPS. If the AC isn’t spitting cold air out the vent by the door, turn the thermostat “OFF”, then to “COLD”, waiting 30 seconds between the toggle to avoid problems with the AC compressor. IF THE BUILDING LOSES PRESSURE TO THE CHILLERS, YOU MUST DO THIS STEP BECAUSE THE INTERLOCKS WILL SHUT OFF THE AC TO PREVENT DAMAGE, AND THE INTERLOCKS ARE RESET BY DOING THIS. BUILDING MANAGEMENT CANNOT HELP YOU WITH THIS – THE SERVER ROOM AC BELONGS TO TB.
- By now, you’re enjoying the fresh air. Make sure all computers are off, and then power on the UPSen. If they fully discharged, they’re going to have a lot of inrush current in the first 10 seconds, so you should do this step. It just might save you a blown breaker. (though I have not experienced this)
- Now we start powering up equipment. Power on all Ethernet switches. They are booted when all the lights are out but power, and maybe a few ports blinking away cheerfully. In 30 minutes you’ll be as happy as those little green lights are, so stay calm.
- Then power on the T1 router. It’s a Cisco 1700 series in the telecom rack. It is functional when you see W0 lit as active. If this is a regional power outage or natural disaster, check the back to make sure the alarm light isn’t illuminated. If it is, the T1 line itself is down. Call MCI/Verizon Business/AT&TMCI/Bell System, American Telephone and Telegraph, and related companies, depending on the level of telecom consolidation when you read this. The circuit is through UUnet as of this writing.
- Then power on the ASA. It is functional when the lights are doing their happy blinky thing.
- Power on DC3 and DC4. This will establish an AD tree relationship with San Mateo/San Francisco/Bunch of people screaming for help floating in the water after the “Big One”.
- Watch them come up on the KVM. Have a beer.
- Power on DC1DT and DC2DT, taking care not to spill your beer on the servers. DC2DT may take a waving of a dead chicken over top of it to properly boot as of this writing.
- Once those come up, make sure all raid enclosures in the office are plugged in and operating. Wave dead chicken as necessary.
- Power on ops-mon, FS3DT, and EX1DT.
- Power on PHONESYSTEM, ensuring both power supplies have power. Green LEDs are illuminated at the back of the power supply, right? This is very heat sensitive, so if you have to delay it to get the server room below 75 F, please do.
- Reset the doorlock controllers. They’re the big box with the ominous triangle printed on them on the wall above the keyboard for the phonesystem. Just yank their power, plug them back in, and listen in amusement as they do a little clicky dance with all the relays. The system is now reset and mostly operational.
- Power on the HP desktop across from the phonesystem. This dinky ass piece of crap is the badging machine. Log into it as local administrator, password is supplied elsewhere. Make sure that stupid dos window with the numbers counting comes up (it’s in the startup group, so it should!). Don’t close that. At this point, you can, in theory, rely on the door locks. Test them at the server room door.
- Power on the backup machine.
- Power on the rest of the computers. I just turn every computer in the room on to be safe.
- As a courtesy, power on the tenant’s crap. You don’t have to, but it builds goodwill and they’re more likely to give you favors. Seriously.
- Check your email. If it works, go home, or whatever.