Backing up entire systems is a very data-intensive process. Not only does it involve moving a ton of data, it requires specific timing in terms of jobs, processes and systems to ensure it is done correctly. Even with dedicated data lines, this process can take hours, or sometimes days, which in general is OK. I, however, found myself in a situation where we had to get a copy of our database from Seattle to Portland by 8am the next day.. And it was already 4pm.
The Problem
We had two data centers that mirrored each other, one in Portland, and another in Seattle. This is a common setup for data centers since it allows for fault tolerance in case one goes offline, and can help provide faster speeds depending on which one you’re closer to. We had just completed a big update in Seattle data center, and needed that update to also be made in Portland. These updates were critical to the success of the project, and due to some (I’m sure very important) technical reasons both data centers had to be up to date. One option was to do this “over the wire”, that is copy/paste the information over the internet. This was the simplest method, however, also the most time-intensive. This was an incredibly large update, and despite having a dedicated line between our data centers estimates ranged from 12-48 hours to complete the update. Unfortunately this delay would break out deployment schedule.
The Solution
Someone pointed out that driving from Portland to Seattle only took about (a very boring) 3 hours, which was a LOT faster than going over the wire. We hatched a plan to copy the updates to a laptop, drive them down to Portland and perform the update overnight. (Despite advances in wifi and internet speeds using the “sneaker net”, or physically walking data from point A to point B is still the fastest way to move data).
Wanting to ensure others on the team could focus on their work (and also to maybe get some quality alone time), I volunteered to be the courier. After quickly downloading some new music I was handed a laptop and a list of commands to run when I got to the Portland datacenter. After a (very boring) 3 hour drive (Protip : bring more than 2 CDs), I plugged into the Portland datacenter and took a nap while the updates ran.
A few hours, and one massive update later, I drove back to Seattle.. Where I promptly crashed in my hotel room for a few hours.
The Takeaway
Ideally projects do not present situations where extreme solutions like driving all throughout the night are needed. Ideally.
When those situations are encountered, however, it’s always a good idea to get the team together to brainstorm. See what possible solutions you can come up with (no matter how insane they sound), and then pare them down to what’s possible given your requirements. In this case, we HAD to have the update live the next day, so that ruled out some options. It was also possible to safely use the sneaker net to make the update happen (e.g. I didn’t have to break all the speed limits).
Plus, sometimes you’ll get a great story out of it.