#138: Good Thing You Made A Backup

The Story Behind Good Thing You Made A Backup

Something experience has taught me is that no matter how much experience I have, I am still capable of making amateur mistakes. We've all watched in horror as our favorite professional athlete falters like a first season rookie - if you're not a baseball fan I recommend looking up Bill Buckner.

One day in early in 2021 I received a pop of orders. If you've ever worked for or run a small business you'll be familiar with the random rush of 4-5 customers seemingly out of nowhere. I've learned, with taleas at least, these micro-rushes are typically the result of a community mention somewhere on the internet: reddit, twitter, a popular facebook page, or other social network. I'm always excited to receive these orders and have made it a habit to personally reach out and thank new customers for their support. Upon reviewing the email addresses for the first few orders I immediately realize these were fraudulent orders. The addresses were something along the line of [email protected]. 🤦 After running ecommerce businesses in one form or another since the 1990's I'm no stranger to attempted fraud, it's definitely part of the business. I promptly log in to my credit card processor, ready to refund the money and report the fraud. To my horror over 500 orders had been placed by the time I was in! In a rush, I quickly turn off my cloud containers and cloud database and then contact my credit card processor to alert them of the orders.

Something that's really cool about cloud databases is that they're always available. The cloud is built to self-heal, so if the underlying infrastructure goes down you don't have to worry about a complete meltdown. Wait a few seconds, and everything is as it was. Magic. Unfortunately, this magic doesn't work as well when you permanently delete a database cluster - including the 7-day snapshots - in a panicked rush. It took me over six weeks to piece together my old website from very old database backups, snapshots from the Wayback Machine, and nuggets I found laying around the internet. I wish I could say that I learned my lesson; but as experience has taught me, no matter how much experience I have, I am still capable of making amateur mistakes.

Plain Text Description of Good Thing You Made A Backup

A server rack full of equipment sits in the middle of a room engulfed in flames. One stick figure, with hands raised, confidently exclaims, "Good thing you made a backup!". A second stick figure, with hands in pockets replies, "I did?"