When the Bear & Giraffe LLC blog started, I set a goal to post at least once a week. I have a reminder set on #Slack for Monday morning to remind me. Setting a schedule forces me to stick to it and not let fall by the waste side like has happened with my many previous attempts at blogging. I’ve been mostly good at keeping up with it. It doesn’t have to be just me who posts, though, Naghmeh Shirazi (or Naghmeh Shirazi, why do you have 2 usernames Naghmeh?) has written a couple of posts for the blog.
I did miss last week’s schedule to post something and honestly, if I had the hour needed to write anything, I would have used it to do something else. Because the last month or so has been non stop for me. It all culminated in two events, back to back. The #ATXHack4Change that took place on the weekend of June 2-4, and a launch of a product on the Monday of the 5th.
I went to the #ATXHack4Change primarily to support a friend, Veronica from The Better Beehive Project (you might remember her from the blog I wrote earlier this year), originally I planned to just be there for moral support, but I ended up actually hacking on an MVP for her. I will share more details about that in a separate post.
So what better way to follow up an all weekend hackathon, other than with a product launch, right? Well, how about a product launch that you’re absolutely not ready for 🤔
The team and I have been working tirelessly over the past couple of months on getting a product ready for launch. I won’t mention which product it is because we’d still like to keep it on the DL for now. It was a soft launch in sorts, and I will publicize it once it’s time. The tech team had been working on building the set of featured required to launch this thing, and if it were up to us, we’d wait another month, maybe a year, before we would launch. It was easily one of those scenarios where if we had more time there would always be one more thing that we would add or fix. But “business” wanted to set a date and stick to it, and I said to myself “Fuck it, let’s do it!”.
I knew we would be building new features that were required for launch up until the last moment. Which is a big risk because we weren’t giving ourselves time to test and validate the deliverables. We also did not give ourselves any breathing room to prepare for launch. In fact, the launch was set for Friday, and I had to make the call to push it till Monday because it became clear a couple of days prior that Friday was not doable, and it’s Friday, you really don’t want to launch on a Friday. Yet I was ready to break that rule too.
I was really hoping that we’d have time to do a dry run. Prepare everything that was needed for launch prior to the day itself. Then on the day of the launch, just click the proverbial launch button, and then celebrate. Typically, the nice thing about web development is that launching is as easy as a simple git push, and even with a platform like Heroku, clicking the “Promote to production” button from the staging environment is all it takes. But our case was not quite just that.
See, the app had been technically live for many months now. We’ve had internal “admins” populating the app with data and uploading pictures of places and whatnot. It was all part of the effort of the entire team to get ready for launch. As we went along and started building things up, we realized that the original architecture had flaws in the way we’re handling the image uploads, and we needed to change how we’re storing the images on S3. Which meant that we needed to move all of the stored images on S3 to a new directory structure, and update all of the references to the files in our app. This is one of those things that would have been prime for a dry run and be done in prep for launch, not at the moment of. But because we were tight on time, and we decided to just move along with the launch regardless of what happens, it became the biggest risk that we had to deal with. So we had this task to move all these files, and update the app, and we had never tested the script that was handling them because our dev environment was not setup to work with S3 but instead to upload the images to the tmp directory of the app locally. So what could go wrong?
Surprisingly enough, the script worked fine, until Heroku timed out on us mid way through (or maybe 3/4 of the way through, we don’t know) and now we had to deal with that situation. Except we had no way of finding out which files moved and which didn’t, because we were copying the files and not moving in order to keep the originals intact, and we forgot to track in some other way the status of each one. So now we had a situation where we were trying to update the app with the new location of the files except we had a bunch that hadn’t moved to their new location and the script was blowing up because the connection to S3 was raising an exception each time it wasn’t finding a file. Awesomeness!!
I did find a way to move the files that hadn’t moved each time the script raised an exception, but because we weren’t tracking the status, the script had to be rerun from the beginning each time. I was able to add a check to make sure that if the app was updated with the new location to skip this file and move on, but that still meant that we had to check each item at a time, and with thousands of records, that took a while.
As all of this was happening, we had to make some last minute fixes and changes to the code. As Murphy said “Anything that can go wrong will go wrong.”, well it did for us too. In the middle of all the scrambling we had to do, our CI suite starting failing for a reason that we couldn’t figure out a fix for. We had 3 apps trying to install a gem from a private github repo, 2 of them were able to, but the third was failing. Everything matched in terms of settings and access and whatnot. So now we had what was supposed to be our sanity check and fallback giving us a “false negative” and we had no way to fix it in that moment. On top of which, since we were making last minute changes and we were in a hurry to get this thing launched, we neglected to run the tests locally. I found out the next day that I broke some tests with some of the changes that I made.
That day, I threw out all of my rules, all of the lessons that I had learned throughout my career, all of the best practices that I preach all of the time. That day was the opposite of an ideal scenario, or even a reasonable scenario. We were supposed to launch at 3PM, we launched at around 6:30PM. I was supposed to be at the Alamo Draft House on South Lamar before 7PM to watch Wonder Woman with my wife (after postponing from the Friday prior because of the hackathon), I got there at 7:20PM and they don’t let you in after the feature film starts, so we didn’t get to watch it that day. But honestly, that was the worst of it, and we got a refund and got to watch it later that week. Nothing caught fire. A few things broke, and we got to fix them the next day. Yet, as everyone else is concerned, the launch was successful, investors were happy, the founders were ecstatic, and I personally was relieved. It’s done!
My lesson learned from that experience is that no matter what you do “the best-laid plans of mice and men often go awry”, as long as you are prepared to deal with what comes up, you will be OK. There are rules and best practices that you can follow to ensure a smooth operation, but if you throw them all out of the window, well you might just be OK anyway. I don’t recommend this path often, but if you find yourself in this situation, do not panic, no one will die, it’s not the end of the world, you’ll get through it 🙂