Burning off the good will points...

[xthread]

So, I seriously screwed up at work the other night,
and basically caused a severity 1 outage. If I did
not already have a phenomenal amount of goodwill
stored up with the customer solutions and
sales teams, I'd be discussing other contract
opportunities with Mindsource.

Luckily, I did have a phenomenal amount
of goodwill lying around, so I'm merely having to
do the penance of documenting everything that I
should have done differently, and everything we
need to change to prevent a similar failure in the
future. But its still icky.

Comments

[d3l1r1um.livejournal.com]

Eeeeek! I'm sorry. I'm glad the result isn't as dire as it might've been, but it sucks regardless. *hugs*

[kyriani.livejournal.com]

Eewww *HUGS!* :( Thats basically my worst work nightmare. Bleh.

[generalist.livejournal.com]

It happens. When you're working for a true 24x7x365 outfit, there will be screw-ups that cause service outages. Sometimes they'll be caused by you. The company's goal is to get the total service-outage time to zero seconds per year, every year—good luck. The reason that you've been getting brownie points is that the company must think you're working really hard to make the customers happy. That you caused an outage just now affects that perception not at all, though it may affect the company's perception of whether what you do is always right.

-(Cheers) generalist

[xthread.livejournal.com]

Heh. The reason that I had rafts of brownie
points lying around spare is that I've been
doing a very efficient job of making the
customers happy (Usually through the
simple tactic of just giving them what they
asked for instead of what I think they should
want. See, at least I can do it in a professional
context).

And of course, the downside of having scary
high-powered troubleshooters is that when
they miss, bad shit happens.

[xeger.livejournal.com]

Heh. Yup. It's amazing how blase you can get about "You know there are millions [or billions] of dollars being lost for every minute that you work on this problem", and how fucking paranoid you get about changes, debugging, and the like.

It was -really- freaky today to take down a server, upgrade glibc, and generally do things I consider vastly irresponsible to do during the work day...