Here is an interesting account of how a company successfully defended itself against a massive DDoS attack in 2014. I have provided my thoughts and recommendations based on the story below:

Executive Summary of the Incident

The account was written from the perspective of a web/infrastructure service provider for a healthcare-related firm. This firm experienced a massive Distributed Denial of Service (DDoS) attack right before its annual client meet up with over 12,000 attendees expected. It was obvious the attackers meant to use the attack as a professional embarrassment to the company.

The firm’s website infrastructure was hosted on a public cloud – Amazon AWS in this case. This gave the firm some strategic advantages in defending against the attack.

The first wave of the attack was targeted directly at IP addresses of the web servers instead of (DNS) domain names. This attack was easy to defend by simply changing the Elastic IPs associated with the domain names. Because the website is hosted on a public cloud, changing IP addresses is very easily accomplished. The cloud provider infrastructure is already set up for making changes like this with no user impact.

The second and longer wave of attack targeted DNS domain names of the web servers, so the first defense strategy could not be used in this instance. Two major approaches were used to defend against this second wave of attack. First, the firm implemented a Web Application Firewall (WAF) using HAProxy. This allowed the firm to use a stricter set of rules to define who is allowed to get through to the webservers. Second, the firm implemented autoscaling for the HAProxy servers as well as the web servers. This in effect, gave them ‘unlimited’ capacity to absorb additional traffic.

With auto-scaling turned on, the strategy was to simply absorb all traffic the hackers could possibly throw at the firm. As the attackers scaled up the attacks, the clients scaled up their servers to absorb the attacks.  At some point, the firm was handling 20gbps of traffic which is approximately 40 times the average size of hack attacks in 2014.

Ultimately, the attackers gave up when they realized that the firm was capable and willing to scale up to any amount of traffic the attackers were willing/able to throw at them. And all this was handled without any significant degradation in user experience on the client’s website.

The attack ended almost in the same manner it began. The attack traffic just stopped and monitoring showed that traffic levels went back to historical normal. Another day, another DDoS attack vanquished.

Additional Recommendation and Thoughts

– The timing of the attack is a cause for concern and unfortunately not unusual. CIOs and CISOs need to be cognizant of the fact that they’re susceptible to attack whenever they roll out new/major products, have scheduled client conferences or expect significant positive press coverage. There’s no shortage of bad actors looking to embarass firms in the middle of new successes.

– The defenses that the firm used are not unique to Amazon AWS. Most public cloud providers offer auto-scaling today. It may be tough to replicate this with a private cloud unless the private cloud has a very large pool of under-utilized servers that can be used to auto-scale.

– The company CIO acknowledged that if they had been hosted on a traditional data center, they would have been helpless as they would not be able to scale to match the attack. The unlimited capacity management native to cloud deployments saved the day for the client’s web user experience during this event.

– Cost of defending against the attack in this instance was a paltry $1,500.00 for over 50 additional large-scale compute servers spun up to handle increase in web traffic. This flies in the face of the general belief that DDoS typically causes a massive spike in fees paid to cloud platform providers. In addition, it’s been my experience that Amazon AWS and MS Azure will work with clients whenever they face DDoS attacks and will possibly waive the additional fees incurred in fending off DDoS attacks.

 – The firm used AWS Cloudfront CDN which has some intrinsic scrubbing capabilities. There are other CDN providers that could have been used as well. e.g. CloudFlare or Akamai Prolexic could have been deployed and also used to scrub traffic before getting to the client’s infrastructure on AWS.