Burning off the good will points...

[xthread]

So, I seriously screwed up at work the other night,
and basically caused a severity 1 outage. If I did
not already have a phenomenal amount of goodwill
stored up with the customer solutions and
sales teams, I'd be discussing other contract
opportunities with Mindsource.

Luckily, I did have a phenomenal amount
of goodwill lying around, so I'm merely having to
do the penance of documenting everything that I
should have done differently, and everything we
need to change to prevent a similar failure in the
future. But its still icky.

Comments

[xeger.livejournal.com]

Heh. Yup. It's amazing how blase you can get about "You know there are millions [or billions] of dollars being lost for every minute that you work on this problem", and how fucking paranoid you get about changes, debugging, and the like.

It was -really- freaky today to take down a server, upgrade glibc, and generally do things I consider vastly irresponsible to do during the work day...