Almost there...

Tuesday, April 14, 2009

Amazon - Sucks To Be You

So there's a lot of talk about the AmazonFail this weekend. As one of those "IT People" who "should've had checks in place to prevent this" so oft cited in comments, let me give a little insight into how these things happen. Because oh yes, they've happened to me, and there's that horrible sinking feeling when you realize what has happened and hope and pray that maybe, just maybe, a miracle can happen and you can fix this before things explode, and then you realize it's way too late.

There's the honeymoon phase where you've got one report and you say, "Hunh. That's weird. Well let me just fix that for you." And then maybe someone else in tech support sees another one. And then most of a week later at the staff meeting someone says, "I had the weirdest thing this week. Has anyone else seen this?" But then that's the day that that other guy is out and then another few days elapse and then you're munching a bagel and someone else comes up to you in the kitchen and says, "Hey, didn't you ask about that weird thing at the staff meeting last week? I just had a case like that..." And you get that feeling. And then you start going down the rabbit hole, and you finally figure out what's going on at 4pm on Good Friday when every Catholic took the day off and all the Jews have gone home for Passover, and you fire off the bug report and pray that things don't blow up while everyone is away for the holiday weekend.

From what it sounds like, Amazon followed the right ITIL protocols on this, but once the world has gotten wind, things spiral out of control pretty quick and there's no way to fix it as fast as things unravel.

We've had this happen with little tiny insignificant changes to PeopleSoft at Stanford or little changes to the Remedy system or little changes here and there. And when it happens, it's bad. When it happens on a holiday weekend, it's real bad.

I feel for the folks at Amazon, some of whom probably spent the weekend driving to and from Grandma's house and didn't hear anything til they got to work on Monday and landed squarely in the middle of a shit-storm before even getting a cup of coffee. That's just never fun. Hopefully it's mostly on it's way to fixed by now.


  • Can I link to this, and the links in the blog?


    By Blogger 815 El Paso St, at 4:13 PM  

  • Sure? (Ryan who?)

    By Blogger Ammy, at 5:14 PM  

  • This comment has been removed by the author.

    By Blogger Chrisfs, at 9:48 PM  

  • Mistakes happen. working with our donor database, I can see how some sort of mass update could go wrong.

    Here's another interesting theory on what happened.

    By Blogger Chrisfs, at 9:57 PM  

Post a Comment

<< Home