When news coverage on a DDoS creates a DDoS - An Analysis of the memcached DDoS on github.com

On March 1^st 2018 several media reports came out stating that GitHub was hit by a missive DDoS attack on February 28^th. The attack was executed using an amplification attack as shown by the figure to the right. In an amplification attack an adversary (A) sends packets to servers (B and C) using spoofed IP addresses belonging to the victim (V). The servers (B and C) receiving the spoofed packets will send the responses to the victim (V). If the responses are larger than the packets sent by the adversary, an amplification is created. The amplification allows an adversary with little resources to generate high amounts of traffic towards the victim. This high amount of traffic saturates the link of the victim rending the services hosted by the victim unreachable.
In the case of the attack on GitHub the adversaries used the memcached service to perform an amplification attack. The memcached service was introduced to speed up the fetching of resources. Retrieving data from disk is usually slow, therefore the memcached service uses system memory on servers to store frequently used data. This speeds up the request time as the data can now be retrieved from fast RAM instead of disk.
The protocol lends itself well for being used in amplification attacks for three reasons: first, there is no authentication required, and second, memcached servers may be accessed by TCP and UDP. As UDP does not require the completion of a handshake, it is trivially possible to forge the IP address of a UDP request, which means tha data from the memcached service can be requested by any entity and delivered to an arbitrary destination. Finally, as the data requests are small and would result in very large answers, the potential amplification factor of the memcached protocol is high, it is possible to trigger downloads tens of thousands of times the original request size. Leveraging servers running the memcached service allowed the attackers to generate an unprecedented traffic surge of 1.35 Terabit per second towards GitHub.

As stated in the title of this article, the DDoS on github.com is interesting because of its new exploitation vector and massive volume, but also because the media attention of the attack created a surge in activity itself towards the memcached service.

In order for adversaries to abuse servers running the memcached service in an attack, they first need to know where these servers are. The obvious way to accomplish this is by scanning the Internet, that is probing each IP on the Internet and see if the servers behind the IP is responds to a memcached protocol request. As adversaries would most likely scan the entire Internet looking for vulnerable instances and use their own IP address to collect the responses, it is possible based on this probing activity to track who is looking for the service and get a fingerprint of by their exploitation attempt. In this study, we leveraged a large unused IP space and collected all traffic destined for potential memcached servers towards addresses in said space.

The figure on top shows the probing activity measured in number of unique IP addresses probing for servers running the memcached service between August and April. By studying the number of unique IP addresses probing for servers running the memcached protocol an estimate can be made for adversaries probing for potential servers to abuse. The red line on the plot indicates the time of the attack on GitHub, 28^th of February, one day before media reports.
The graph shows that prior to the media coverage, actually little scanning activity can be seen. However, after the media has covered the attack, the activity suddenly increases from a few unique IPs to between 25 and 50 IPs scanning per hour for vulnerable instances. One exception can be noted at the end of January when around 125 unique IP addresses scan for memcached servers. From the figure it seems that the media reports have triggered an increase in activity. Likely adversaries trying to uncover vulnerable server which can be used in future attacks.

The number of unique IP addresses probing for memcached servers is a good indication of activity. However, it only tells a part of the story, it does not show the intensity of the activity. We can measure intensity in terms of probes received per hour, after all the faster the attacker scans the Internet, the sooner a sizeable list of exploitable services will be available. Given the size of our monitoring IP ranges, we can obtain a solid estimate of the scanning speed of each actor.
The figure on the left shows the intensity of probing activity, measured in packets received per hour. It is similar to the figure portraying the unique IPs per hour, showing the same increase, in terms of packets per hour, around the time of the attack on GitHub. However, contrary to the previous figure, there are four distinct intensity spikes prior to the attack. This indicates that prior to the attack, some reconnaissance missions are performed to find servers which can be used in an attack. The spike in intensity of late January coincides with the spike in number of unique IP addresses, the three spikes prior do not. The discrepancy between activity and intensity give an insight into another aspect of adversaries and probing techniques.
The three first spikes are created by one IP address, the fourth one is generated by 125 different IP addresses from the same /24 subnet working together. The use of multiple IP addresses shows that there is a difference in adversary skills or resources. Using multiple IP addresses is more costly and complicated than using a single IP for probing.
Based on the scan progression, we can see several reconnaissance campaigns prior to the attack. Additionally, it reveals that some adversaries are more skilled or possess more resources than others.

When we look at the payload of the probing packets, we can obtain further information about the adversaries, for example in how they structure their packets, whether they make mistakes in doing so, or how they optimize their techniques to increase the impact. Indeed, analyzing the different payload of the probes reveals that there is a plethora of different probing packets, indicating different knowledge and skill levels. Some adversaries manage to immediately craft correct probing packets whereas other take several attempts to make valid probes.
The difference in payload and how the packets are formed also allows some fingerprinting of the tools used for the probing. Some probe types are specific to single IP addresses whereas othes gain fast some wide-spread adoption. As shown on the right, the majority of senders finally converge to the "correct" solution, which will trigger a sufficiently large response from any unsecured memcached server without a mistake in the request packet format. This development seems greatly helped with the publication of two proof-of-concept programs on pastebin, that demonstrate how to send a valid packet in the required format. Shortly after each PoC publication, we see an influx in people participating in memcached probing and a spike in more and more actors adopting this technique. The progression towards the right solution also shows us that some adversaries are more skilled than others, and shows clear evidence for the collaboration between IP addresses.

For a more in depth read please refer to the original publication: How Media Reports Trigger Copycats: An Analysis of the Brewing of the Largest Packet Storm to Date. In addition to the increase in activity related to media reports and the difference in skill set the paper analyses the convergence of commands used for probing, the research also analyzes the response of memcached server operators on the events following the media reports.