Disclaimer: although the whole situation around SolarWinds is highly politicized, we focus only on the tech side of the incident in this post. Our goal here as a mobile and web development company is to research the security insights our clients might find handy to protect the data they work with.
On December 13, 2020, Reuters reported that hackers had monitored the internal traffic of the U.S. Treasury and Commerce departments.
The hack became possible by hiding the malicious code in the software updates released by SolarWinds, an IT company serving not only America's Fortune 500 companies but also such government entities as the NSA and the Office of President of the United States. Later, Solarwinds would report that nearly 18,000 of its customers all over the world received that software update. And it was distributed between March and June of 2020.
Investigations following such large-scale cyber incidents may take months or years to complete, but as of February 2021, there are already a few important insights and milestones we’d like to go through.
Sunburst: the Backdoor in Orion DLL
It all started with FireEye's detection of the SunBurst backdoor, a trojanized version of a SolarWinds Orion plug-in able to communicate with third parties via HTTP while masquerading as OIP, the legitimate Orion Improvement Program protocol.
After being initially idle for two weeks, the backdoor starts transferring and executing files, profiling the system. Other abilities include rebooting the system and disabling its systems.
As FireEye – another victim of SunBurst – was researching its samples, they found Teardrop, a memory only dropper that runs as a service. None of its parts had ever been seen in any other malware. But we'll get to it later.
There are multiple reasons why Sunburst wasn't detected early:
- Sunburst used obfuscated blocklists to avoid anti-virus and forensic tools able to stop it
- It set hostnames matching legitimate hostnames of the environment, which made it harder to notice, though IP scan history could’ve revealed unusual IP switching.
- The attacker used the IP addresses of the same country as the victim, complicating noticing the unusual traffic. It is possible to notice multiple logins into the compromised accounts in geographically different and unusual places, but only if you’re looking for exactly that in the login/IP logs.
- Multiple compromised accounts were used to navigate the infrastructure. That can also be noticed by specific software, but it's a scenario too uncommon for most attacks.
- Legitimate files and tasks might be temporarily replaced to execute necessary utilities and the only way to notice it is to monitor logs for delete-create-execute-delete-create patterns of actions that happen suspiciously quickly.
As a result of this investigation, FireEye released a git repo to help others detect SunBurst.
According to Charles Carmakal, senior VP and CTO at Mandiant, which is an incident response department at FireEye, SunBurst was detected because they discovered an unauthorized device in the multifactor authentication system.
Their employees use it to enter the system remotely via the company's VPN. As a part of the login procedure, the employee must enter the code sent to their mobile phone. The hackers registered such a device to get unique codes, but it caused the security system at FireEye to issue an automatic alert and send it to the security team.
Although the attack was stopped, it meant that at that point the attacker already had the username and password credentials. Investigating this, FireEye eventually revealed the SolarWinds breach, using which the intruder got into the FireEye network and retrieved necessary credentials.
On December 18, the Yahoo News reporter Kim Zetter investigated that in October 2019, the adversaries had distributed the malware in the same way, but the files did not contain the backdoor to provide hackers with direct access to networks. The malicious code wasn't executed. The reporter's sources told that this was a dry run for hackers to see if such a method would be noticed and compromised. Later SolarWinds confirmed this info, adding that the earliest suspicious activity was identified in September 2019.
As to how the SolarWinds updates were injected with malicious software – we don't have info on that. But in November 2019 a bug bounty hunter Vinoth Kumar found a public GitHub repository leaking FTP credentials belonging to SolarWinds. Any penetration testing company would agree hackers could have easily found and used that – among other possible breaches – to deliver Sunburst, though the file with the injected code appeared to have a legitimate digital signature, so the intruders needed more than just access to the filesystem.
Sunspot: the Implant Launching Sunburst
On January 11, CrowdStrike, the company helping SolarWinds investigate their supply chain attacks, shared its research on the Sunspot implant. This is how they called another malware that was deployed in September 2019 and used as early as February 2020 to insert Sunburst into the builds of Orion.
The implant monitored processes related to the compilation of the Orion product, and at a certain point replaced one of the source files to inject Sunburst. Moreover, it included code to disable possible warnings and perform modified MD5 checks to not let the Orion build fail and alert developers about a possible presence of an adversary.
Along with that, Sunspot's source code files were encrypted, its log files were encrypted and masqueraded as VMWare logs, and the entries inside of those files were also not easily readable thanks to a hard-coded separator value.
Raindrop and Teardrop: the Access Granting Malware
On January 18, Symantec uncovered Raindrop, another piece of malware related to the SolarWind attacks, that, however, was only used against specific victims.
Raindrop is similar to the above-mentioned Teardrop, but unlike the latter, it spreads across the network and there is no evidence it is initially delivered there by Sunburst. So while Teardrop is used when the machine is infected with Sunburst, Raindrop helps infect other machines on the network and is most probably delivered there manually.
The ultimate goal of both malware is to deploy Cobalt Strike Beacon, a thing we as also a quality assurance/testing company know all too well.
Cobalt Strike: the Security Tool That Got In Wrong Hands
Cobalt Strike is a legitimate framework with an easy-to-use interface for post-exploitation. It emulates an adversary actor in your network to help you improve your incident response and security operations via penetration testing.
Although Cobalt Strike is mainly distributed among security researchers, there were reports hackers use it too.
The goal of Raindrop and Teardrop is to release and distribute Cobalt Strike Beacon, a client agent that is executed in memory to leave minimal footprints on the disk. As soon as an intruder has a connection with Beacon in the victim environment, it can leverage it to open remote access and execution.
What's interesting is that every Cobalt Strike DLL implant was completely unique for every machine: no overlap or reuse of folder/file/function names, IP addresses, requests, metadata, or processes involved. The same was applied even to non-executable entities – all to prevent identification of compromised assets or threat intel sharing between victims involved. (via Microsoft).
Avsvmcloud[.]com: Key Node Between Attacker and Victim
After Sunburst would remain passive for the period of up to two weeks, it would then ping a dynamically generated subdomain of avsvmcloud[.]com.
The subdomain would reply with a DNS response containing a CNAME field including the eventual domain from where Sunburst would obtain instructions and payloads to execute on the network. That means that data leakage would be happening through the DNS traffic which is usually used for domain resolving and thus would probably be ignored in terms of security. Until now.
On December 15, 2020, a coalition of tech companies – FireEye, Microsoft, and GoDaddy – sinkholed the initial domain. Now it is in Microsoft's possession. Sinkholing allowed security researchers to create a list of infected victims (with their systems trying to connect to the domain) and report the incident to them.
Along with that, the domain is now turned into a killswitch preventing Sunburst from further operation and thus deactivating it. However, that doesn't remove the backdoor and still allows the adversary to leverage it, though now it'd much more difficult to do that.
SuperNova: a Next-Gen Webshell
Along with initial findings, there was one more Orion artifact dubbed SuperNova. It's a trojanized copy of a SolarWinds .NET library exposing an HTTP API to supply the SolarWinds Orion app elements with its current logo on request.
In this hacking scheme, SuperNova is probably its most advanced part. Palo Alto Networks and Guidepoint Security researched it and here is what we learned from them.
Unlike Sunburst, this file wasn't digitally signed, implying that it was dropped by a third party in the middle of the attack instead of being implanted beforehand.
Usual webshell attacks aimed for surface-level exploitation. SuperNova is nothing like that: it's a stealthy, full-fledged .NET API that works with .NET programs as parameters.
While a traditional webshell would leave traces on the disk and in the network logs, SuperNova works more like a launcher compiling and executing full-featured programs written in C# on the fly in memory. And its code also minimizes the risks of scrutiny by preventing error report generation.
Multifactor Authentication Bypass
On December 14, 2020, safety agency Volexity reported that the same attackers had penetrated their network no less than three times between late 2019 and early 2020.
And though the SolarWinds breach was leveraged only in the third attack, it’s worth noticing how the same intruders bypassed MFA in the second attack.
Volexity uses Cisco's DUO for multifactor authentication protection. Once the intruders used the backdoor to get administrator privileges, they made the memory dump of the Outlook Web App server to access the Duo integration secret key (aka "akey"). With this key, they generated the so-called duo-sid cookie. After logging in to the system with stolen credentials, the Duo server evaluated the cookie as a legitimate one and didn't request an additional identity check.
However, from what was told by Volexity, it seems like in this way the intruders were able only to access the e-mail account of a person whose credentials were hijacked.
(It might also be important to add that no vulnerabilities of Cisco's Duo were found and leveraged in this scenario according to Duo officials.)
Latest Status (as of February 2021)
On January 12, a site named solarleaks.net was found on the web, selling the data that was allegedly stolen during the SolarWinds attacks. The offers include source code data of SolarWinds, FireEye, Microsoft, and Cisco, altogether offered for $1M. It hasn't yet been confirmed that the site is legitimate.
On January 19, 2021, Malwarebytes reported another intrusion vector by the same threat actor. The company does not use any of the SolarWinds products, and the threat they noticed in December was coming from a third-party app in Microsoft Office 365 that behaved in a way similar to the malware used in SolarWinds attacks.
It was possible for an intruder to place it there thanks to a flaw with Azure Active Directory found as early as 2019. The flaw was allowing backdoor access to principal's credentials, and eventually, CrowdStrike released a tool to mitigate risks connected with it.
No internal systems suffered from unauthorized access – the adversaries could only read a limited number of internal emails.
Big Picture and Bottom Line
Having so many details, we can now visualize a possible attack scheme.
With this scheme, you can imagine how sophisticated the attack was.
As our DevOps company department was studying it, we realized that for most companies there was no way to spot it at the very beginning: you need to either have a dedicated DevOps team watching the logs and having cohesive monitoring systems up-to-date at all times, or limit your infrastructure setup in a way that, say, prohibits using any external servers for any purposes.
Otherwise, if you depend on third-party software, which is a case all too common, the best thing you can do is to outsource the protection of your digital assets to a specialized security firm. Not because they are able to protect you, but because they can spot and repel the attack in time – just like it happened with FireEye, which noticed a suspicious device authorization in their MFA system and prohibited the intruder from going deeper.
Initially, we wanted this blog post to be a deep dive into the sophisticated security practices helping to avoid becoming a victim of such attacks. Yet it appears that the best things you can do come down to the next shortlist:
- Keep an eye on your network. Manually. Attacks like SolarWinds one bypass vulnerability scans and defense software, so you're left to continuously watch after your network with your own eyes.
- Study your outbound traffic beforehand. Suspicious changes might be spotted only when you know what is not suspicious in your network.
- Practice air gapping. This means physically isolating elements not requiring the connection with the environment – devices, data, etc.
- Follow Zero Trust strategy. Since most of the attacks are trojanized and come from inside, it's best to verify the credentials of every entity – a user, an IP address, a device – trying to access your resources.
- Harden your system. Remove non-essential software on every device so that it would be used only for what it is intended for.
Stay safe and secured at all times.
