No OOB

Chill Uncategorized

Ahahaha! This is not normal. This tells me that Facebook does not have any out-of-band management set up. That’s a way only “smart” people fail. “Out of band management” is some other method of getting into your vital systems that doesn’t depend directly on those systems themselves. Every good-for-anything networking expert sets such things up because you never know when something like the above will happen. For instance, I’d have a completely separate IP KVM connected to a console port on Cisco switches that in no way depended on those switches.

But of course because they are the soi-disant “smartest people in the room” I am sure they thought they’d never need any form of OOB and thus never bothered. That’s how people like that always fail.

If they’d had OOB, it could’ve been a 15 minute problem instead of a 15 hour problem. Or at least a whole hell of a lot shorter.

(By they way, if you don’t have OOB, something else you can do is write the commands in script form, and put a two minute timer at the end that reverses all the changes unless you kill the script. That way if whatever you’re doing kicks you out of the system and brings it all down, your clever little script automatically reverts back to the original after two minutes, letting you back in. I have done this before too and it has absolutely saved my ass.)