A whole week might have passed since Halloween, but for some reason, we’re still in a spooky mode!
After our special Halloween post about the 7 deadly developer job listings sins, we think it’s time we gave you that good old Halloween scare! Here are the top 4 DevOps horror stories that will make you run for the hills!
#1: The revenge of the backup
More than a year ago, Gitlab experienced an epic whoopsie by accidentally deleting production data while performing routine database maintenance. But that shouldn’t be such a big deal now, would it? After all, this is what backups are for… right? Well, not really if your previous backup procedures had not executed correctly or you haven’t ever tested them because, you know, you had faith in the backup going off without a hitch!
Moral of the story: Routinely check your backups, people!
#2: You never imagined failed deployment could be *that* costly
This one is quite famous actually. Back in 2014, a company called Knight Capital went bankrupt after losing some $440 million in 45 minutes and disrupting the stock market. Cause of death: Failed deployment! It all started with some technical debt, 8 years old legacy code that hadn’t been used since God knows when, and an unstable delivery procedure, new automated trading code made it on to 7 out of 8 SMARS servers.
However, the eighth server still had that legacy code. And the time came when a repurposed configuration value activated the eighth server, and it began making automated trades at such speed as if the Flash was operating it, causing a disruption in the prices of hundreds of stocks and moving millions of shares.
#3: Expectations vs. reality
This Hackernews commenter could not put it more accurately!
Expectation – Monday Morning: T1 install will be complete. Tuesday: Test/bootup period. Wednesday: Sales start Thursday: Sales continue, TV ad goes live Friday: Champagne!
Reality – Monday Morning: T1 did not get installed. Tuesday: Emergency ISDN solution (stolen from Chiropractors next door) Wednesday: Modem rack catches fire Thursday: TV ad goes Live Friday: T1 goes live. Champagne.
#4: It’s almost weekend… or maybe not
This is one of those one-sentence horror stories that will inevitably give you the chills:
Someone deployed new features on a Friday at 5pm. The end.
Stop that madness!
Do you have the feeling that you would have had one or two things to say about tech-related horror stories if you simply had the chance? Well, it’s your lucky day! The call for papers for JAX DevOps 2019 is wide open!
We want to hear from you, a group of diverse, intelligent, and really friendly DevOps engineers, scientists, and more. Next year’s JAX DevOps takes place in London May 14th – 17th, 2019. Is your schedule free to present a paper on that interesting thing you’ve been working on? Here are this year’s topics:
- Business Company Culture
- Cloud Platforms & Serverless
- Docker & Kubernetes
- Continuous Delivery
- Microservices
- Monitoring & Diagnostics
The deadline for submissions is November 12th, 2018! Submit your proposal today!
The post Top 4 DevOps stories from hell appeared first on JAXenter.
Source : JAXenter
