DevOps Horror Stories: Repository of Horror
The Repository of Horror, is a series of three short, self-contained, software-related, horror stories. Read them if you dare!
Just when you thought it was safe to go back in the water...
Is there anything more frightening than the unknown? Anything the mind can conjure up is frequently scarier than something realized. The shark in Jaws is terrifying because you don’t see it until it’s too late. It’s a silent, relentless death machine, hiding in the water.
A software vulnerability is the unknown, hidden deep within an ocean of code, packages and container dependencies. Once exposed, you’ve no protection, you may have already lost data, or given someone continued access to sensitive information or processes. Even worse, do you know if you are exposed?
There’s somebody at the door
Software, by its very construction is like a boarded up wooden house in a Zombie movie. You’re trapped, zombies are roaming around outside trying to get in, clawing at the doors and windows, ripping off hastily nailed-on boards, creeping under the floors in the crawlspace, searching for an opening. It’s a living (dead) nightmare.
Why would you want to be the one tasked with securing the house, patching the holes, letting the frightened neighbors in? Who wants the responsibility of ensuring everyone's survival?
The Butterfly Effect
It happens to all of us eventually. You make one tiny change. The PR is approved. It passes QA. It gets rolled out to staging. Everything looks healthy. It’s good to go. It gets rolled out to production late on a Friday afternoon. It's a simple change. What could go wrong?
A few moments later, the app seems slow. The processing queues are backed up a little, but are within the margins. A warning pops up that you haven’t seen before. You start to investigate. A few minutes later, the first error is flagged. A few moments after that, the first customer asks if there is anything wrong. Then the second.
Suddenly, the database is running hot. The queues are backed up beyond acceptable levels. Something is broken. Badly broken. But it just couldn’t be that simple change, could it?
Deep down, in the darkness and silence, you know the truth. These problems are systemic and common. You need to address these concerns; a control plane for security vulnerabilities, a way to quell the total cost of ownership of running a key part of your automated workflows, and the assurance of building repeatable DevOps processes to make sure that final deployment is trivial and quick to rollback if needed. You need a weapon built for the times.
Liked this article? Don\'t be selfish (:-), share with others: Tweet