Almost there...

Friday, June 10, 2011


I've been working in IT at Stanford University for over 10 years now. Every year, as Commencement approaches, there's a 2-4 week freeze on everything we do. We can't change anything for fear of breaking something and slowing down the Commencement machine in it's critical hour. Every year, Commencement comes and goes without a hitch.

Today, around noon, one of our email servers got a bad case of data corruption. Early attempts to get it back up and running failed. This was going to take a full restore. That's bad.

The good news is that that server that went down is largely populated by IT Services staff. The bad news is that it's not all IT Services. Turns out, it's also Student Financial Services, the Registrar's Office, and the assistant to the Dean of the Medical School. On the day before Commencement weekend, this is Really Bad.

So, about 3 p.m., I gave my phone more of a workout than it's ever had. I called stakeholders in every area on campus (since email is broken, yo) and talked about who was impacted and we moved critical personnel to an alternate server so that they could read their queued messages and continue working. At 5 p.m., the Dean's assistant's messages still weren't coming through, so fearing a possible PEBCAK issue as she used the unfamiliar Webmail instead of Outlook, I sprinted down to her office to check it out. In the fine tradition of having the IT guy standing over your shoulder, it immediately got with the program and started working.

Now everything is back to normal and I'm be off to dinner and Friday Night Waltz. But for the first time ever, I got to be part of the Commencement madness in a participatory way. As I sprinted out, they were setting up a photo of medical grads. As I came back, they were having one young lady stand on a turned over plastic bin for the photo, but her heels were threatening to collapse the bin. A couple of parents were escorting their obviously beaming son down Via Ortega. There's a giant tent blocking part of the walk way between Clark Center and LKC. It's all very exciting here. And at the end of the weekend, the grads and their families hopefully won't have noticed anything wrong. And once again, I adore our UNIX admin who was technically on vacation today, but has spent the last six hours pulling a rabbit out of her hat. I work with awesome people. We get stuff done.


Post a Comment

<< Home