Work was a major cluster effing

[Captain FT]

One of the unix admins with root access mistakenly typed 'date 2009' instead of 'cal 2009' on the primary autosys server, this was in the early morning. So every job between early morning and 20:09 or 8:09pm starting running because the jobs thought they were behind schedule when the server time warped to 8:09pm

for those who don't know cal brings up a calendar, date changes the time of the server

not a fun day

only unix nerds will appreciate the magnitude of the cluster fucking that shortly ensued

 

[Razorguns]

Time to utilize that 1:1 backup and switch to an east/west server in the meantime right!

r

 

[Captain FT]

Time to utilize that 1:1 backup and switch to an east/west server in the meantime right!

r

its load balanced so we shut down the affected server and the secondary took all the traffic.

 

[binö]