Working Good

Just made a big mistake that affects system operations. Tell me your past mistakes to help me feel less badโ€ฆ

Iโ€™ve been lucky as I havenโ€™t made any really huge mistakes, but if you donโ€™t bring at least part of production down sometimes, youโ€™re not a real sysadmin doing anything consequential.

In my career, Iโ€™ve:

1) Rebooted the wrong server (at least twice, prod instead of dev/test).

2) Locked myself out of a firewall, had to drive on site (luckily only 20 minutes and it was after hours) and connect to the console to fix it.

3) Nuked a DHCP scope that was still valid, meaning those computers had nowhere to renew their leases when it came time, keeping them from accessing the network.

Those are just screw-ups I can remember. I am sure there have been others. But as a sysadmin, ultimately even if you test something youโ€™re making the actual change in prod. And for us, prod is never exactly like the test environment as thatโ€™s just not possible. And no one is perfect. These things will happen. Having quality backups and a plan to revert to a working known good config is key.

Leave a Reply

Your email address will not be published. Required fields are marked *