99% of DDoS-DFIR (primary the forensic part) consist of botnet-analysis and dealing with the "usual suspects" like booter-services and smaller botnets.
But every once in a while we come across an interesting attack like this with attackers who either play tricks with the Os-people or are able to attack efficiently.
as in this case, which we want to report on here, where the attacker:
- shut down the VPN-gateways to deny access for tech and ops
- shut down the intra-datacenter routers (corerouters)
- shut down the authoritative dns-infra that affected all clients
all within minutes.
The attacker used his OSINT-FOO to get the information needed for steps 1 and 2, and was able to achieve his goal through a small but fatal misconfiguration. But more on that later.
Disclaimer: All screenshots were taken with the help of our DDoS analysis platform, with which we are able to analyse log files (firewall, sflow, web/access logs) and evaluate them from a DDoS point of view. For an overall view and assessment of the attacker, it is always important to know: what happened when, how and why, and in what order, especially when you need to know if an attack happened by a scriptkiddie or an advanced attacker.
Prelude // 1st Wave: Attack on the VPN dialins and corerouters
In the first wave, neuralgic points were attacked:
- the VPN gateway for the service staff
- 2 corerouters, which were responsible for the data transfer between the individual locations/datacentres of the company.
With this attack, the service staff was quickly incapacitated, as 70% of the technicians work from home and were dependent on the VPN endpoint.
In addition, "shooting down" the corerouters prevented any intra-DC data traffic, so rerouting and using fallbacks from the backup DC and many other things did not work.
How did the attacker get this information? Insider? crystal ball?
The first impulse would be "insider knowledge".
Well, as we have seen in the State of DDoS-Summary 2021, there are more and more attackers with a professional approach, such as recon and enumeration via OSINT and clever target selection.
Because in this case, attacked systems and approach can be easily clarified by the following assumption:
- it was a targeted attack
- the actual target is to remain disguised
- maximum impact needed
Since there was no blackmail letter or ransomnote, but the attack otherwise did not give the impression of a sledgehammer method, we assumed a targeted attack by an experienced attacker. And experienced attackers know Recon and target selection.
So we performed a mini-recon ourselves, checked all the IPs available online via shodan and then had reverseDNS for all IPs and parsed for "[fw|gw|vpn]", et voila!
In total, we found more than a handful of firewalls and VPN gateways, but these could be assigned to customers rather than directly to the data centre, and a total of 4 from the datacentre itself.
This clarifies the first point, how the attacker could potentially obtain the information regarding the VPN endpoint for technicians.
The second point, how the corerouters used for the inter-DC traffic could be identified, is similarly easy to answer; we will use a traceroute from Hetzner as an example.
Here you can see the EdgeRouter "versatel-gw.hetzner.com", then the intra-DC corerouters between Frankfurt (fra) and Nuremberg (nbg), which shouldnt normally be reached or pinged.
But in the case of the attack we analysed, it did. And so the attacker was able to limit the ability to act for the technical staff and the intra-DC communication in a very short time, headshot!
2nd Wave: Attack on the Authoritative DNS Infrastructure
The second wave hit the DNS infrastructure, and since the datacentre utilizes CNAMES heavily and is the Auth-DNS for more than 30,000 domains, this part of the attack affected all customers.
Successful DDoS attacks on DNS infrastructure always have devastating effects.
Nothing more to say here, except that, due to the 1st wave, technicians couldnt mitigate the attack as fast as execpted.
That Little Misconfig-Detail
Now the real question: why was the attack/sequence of attacks successful? Did the target have no DDoS protection? No alerting, monitoring, etc., no DDoS testing?
The datacentre had a very well set DDoS detection and mitigation, and a good protection provider for the core services (hybrid approach), which can be switched on if necessary. The DDoS appliance also reported the attacks and sent alerts.
We also have regular intense DDoS-Stresstests with the technical staff, and the TimeToMitigate was < 30 seconds for all attacks, from simple to severe.
The crucial point here was:
- The first measure consisted of detection and blackholing of the affected IP.
- The second measure is then manual switching the affected networks via BGP rerouting to the protection provider.
- there was no blackholing-whitelist for the core services!
This last little circumstance ultimately led to the attack being successful for about 30 minutes, despite mitigation actually working well.
Fortunately, there was an experienced senior technician in-house who, through regular DDoS training, immediately understood the problem and its effects and was able to coordinate the correction of the faux-pas and the defence against the attack from within.
After an hour, the spook was over and all systems were working as expected, even though the attacker continued to try his luck for a few more hours.
IP-Spoofing IS A Thing
A small follow-up on the attack method, because we have not yet seen it ourselves, but have already carried it out:
The attack was a fake DNS reflection/amplification flood that used IP-Spoofing to select a different sender address for each DNS packet.
Fake because the actual sender IPs were not OpenResolvers, but already valid IPs from servers or Broadband-AS; so it was not the destination address that was spoofed for a reflection/amplification attack, but the source address.
The IP generation was very intelligent, no bogus IPs appeared. It looked as if the IPs were randomly generated using valid AS/CIDR lists.
Spoofing is supported by the fact that we do not believe we are dealing with a 900k botnet.
Summary: We assume a targeted attack with an above-average attacker (DRS level 5 "Advanced") who wanted to attack a customer of the data centre without pointing to the actual target (smokescreening).
The attack was intelligent and more like snipering than a sledgehammer.
Lessons learned
Main lesson:
-
Carry out mitigation and measures "to the end"; during the last stresstest almost a year ago, the technicians were already aware that the implementation of whitelists was still pending. Unfortunately, this got lost in the day-to-day business and was left undone.
-
DFIR in the aftermath of successful attacks is always useful if the threat situation is not clear.
-
If you want to be on the safe side, you should think about echtes DDoS-RedTeaming
Fragen? Kontakt: info@zero.bs