Every large site, and many small sites, on the internet will get attacked at some point. The reasons vary from something as trivial as a bad comment left on a forum, political beliefs, or even a company trying to damage a competitors business. The goal of these attacks is to shut down the targeted computers, network, or data centers so they can not conduct business or serve their customers.

The recent attack on our provider, Linode, was much, much larger and lasted much, much longer than a typical attack. Attacks of this size tend to be done by some really dangerous people trying to extort money from really big companies. Tap Inspect was one of hundreds of Linode customers that got caught in the crossfire as the battle was waged.

The attackers started with the entire Linode system which includes data centers all over the world. As it was beaten back, the attackers changed tactics and appeared to concentrate on Linode’s Atlanta data center. The home of one of our servers.

From our and our home inspectors point of view this meant it was very tough, if not impossible, to connect to some of the Tap Inspect infrastructure during the height of this battle. The result was many people had to wait several hours until they could publish their reports, sync their devices, or pick up reports from the web. We are so grateful for all your patience.

The mobile app was functional and all our user’s data was safe. We had moved it off of the Linode system months ago as we built a more robust and fault tolerant system. We still had our application server at Linode in their Atlanta data center. This is the server that talks to the mobile app and also displays the pick up pages for reports. It got pounded just like all the other sites in Atlanta.

As the attack moved into New Year’s Day we were doing everything possible. We were manually pulling reports for our users so they could email them directly to their clients. We were answering phone calls, Facebook messages, and support emails and making plans what to do next.

The first Monday after a long holiday was rapidly approaching and our users would need their Tap Inspect to work. We have had a disaster plan in place and knew it could take 24 to 48 hours to bring an entirely new Tap Inspect system up from scratch. Linode was making progress but service was still coming and going. We had to make a decision.

Should we hold on tight in Atlanta? Should we move our server to another Linode data center? Should we abandon Linode completely and hope our new host doesn’t get attacked?

We decided to hope for the best but plan for the worst. Our disaster plan was implemented and we began building a new more robust Tap Inspect. We knew this could take a few days so we also decided to stay put in Atlanta.

As Sunday morning came appeared that Linode may just have won the battle. Our server’s performance had returned close to normal. We didn’t want to tempt fate so we continued with the disaster plan and brought up our new servers just in case.

Now it is Sunday evening and several hours have passed. It seems the worst is over and Monday morning should be a good day. The Atlanta data center is working as expected but we have plan B on deck just in case.

We are so thankful for the brilliance and dedication of the team at Linode, the skill of my Co-Founder, Jason Adams, and the patience and support of all our users.

We stood our ground against the bad guys and made our corner of the internet a little bit safer.