Contents
What is Log4j2 CVE-2021-4428?
What is Vulnerable and Which Version Has The Fix?
What Does a Compromised Application Do?
What Inventory Should I Gather?
What Information Am I Gathering?
How Am I Fixing This?
What Do I Look For?
I Removed JndiLookup.class. How Do I Validate?
I Upgraded to a Fixed Version. How Do I Validate?
Build Process?
What Can My Other Teams Do?
What Else?
Log4j has lit the internet on fire. Understanding, scoping and assessing the situation due to log4j has been vital to remediation and finding a solution to this pervasive problem. In the last 2 weeks, I’ve had the good fortune to work with a talented team to address the problem for our own deployment and to help our customers. This posting is a generic summary of my notes, observations and actions used towards addressing the problem.
In a follow-up post, I’ll summarize how exactly we addressed the problem at Uptycs.
What is Log4j2 CVE-2021-44228?
- One of the most severe security vulnerability in recent times
- Only impacts Java application which are using a very popular logging library (many)
- Application does not need to be Internet facing. If user provided input makes its way to the application and is processed by the logging library, it may be vulnerable
- A well crafted string sent as a part of HTTP header(s) or the HTTP body, if it reaches the logging library can make the application vulnerable
What is Vulnerable and Which Version Has the Fix?
- log4j (1.x) is a legacy library. Even though it has its own vulnerabilities, none as severe as this
- log4j-core jar file from log4j2 library has the vulnerable class
- log4j-core-2.0-beta9.jar or later versions are vulnerable
- log4j-core-2.12.4.jar (for JDK7) and log4j-core-2.17.1.jar (for JDK8 and later) have the fix
What Does A Compromised Application Do?
- Pretty much anything is possible
- Exfiltrate secrets and other sensitive data
- Move laterally to other parts of the infrastructure
- Ransomware/spyware/... attacks
- Exploit does not necessarily create additional processes
- Initial exploit instructions may be downloaded from the Internet by the vulnerable Java application
What Inventory Should I Gather?
- Dev, staging, test, production, ...
- Cloud VMs, containers
- Serverless
- Artifactory and other internal caches
- Container registries
- Blob stores (S3, etc) with build artifacts
- Employee machines
What Information Am I Gathering?
- Vendor software with Java
- Looking for vulnerable Java applications
- Looking for all log4j version 1 and 2 artifacts
How Am I Fixing This?
- Leave log4j 1.x alone? Upgrade? What about third-party software with no option to upgrade to the latest log4j2? (lower priority)
- Upgrade to 2.12.4 (JDK 7) or 2.17.1 (JDK 8 and above)
- Add Environment variable (LOG4J_FORMAT_MSGh_NO_LOOKUPS=true) or JVM property (-Dlog4j2.formatMsgNoLookups=true)? This doesn’t seem to fix the issue completely. Doesn't look like the issue is fully addressed:
- Delete class file? Where is this done? Post build or pre deployment? Post build is ideal
What Do I Look For?
- Java processes
- Containers with Java processes
- JDK/JRE version
- jar/ear/war files. Unfortunately java -jar can run any arbitrary file with any extension. Lets hope you don't do that
- Do I have uber jars, shaded jars, jmod, ...
- Do I have compressed jars etc?
- Are the jars obfuscated?
I Removed JndiLookup.class. How Do I Validate?
- For each Java process, if jps is available:
|
for i in `jps | grep -v Jps | awk '{print $1}'`; do
jcmd $i GC.class_histogram | grep -q JndiLookup && {
echo "PID $i MIGHT be vulnerable";
};
done
|
- Basically make sure the class is not loaded by your application
I Upgraded To A Fixed Version. How Do I Validate?
- Get inventory of all jar/war/war/jmod files opened by the process
- For each file opened by the process, check if it is or contains vulnerable log4j2-core jar
- Validate on on-going basis to avoid deploying vulnerable applications due to auto scaling, old build artifacts etc.
Build Process?
- Scan container images for vulnerable version
- Scan Java artifacts for vulnerable version
What Can My Other Teams Do?
- If you have a Web Application Firewall (WAF), use it. Make sure you apply the following rules:
${.*${ or ${jndi or ${ctx strings matching HTTP headers or body payload
- If you have EDR/XDR product, for non-patched assets/Java processes:
- Monitor processes spawned by Java processes
- Monitor egress traffic to Internet (if the process is not expected to reach out to Internet)
- Now is the time to look at egress firewall/security group rules. If your Java processes are in a subnet that the process is not expected to reach out to Internet, add egress blocking
- Double check your data back-ups and disaster recovery steps
What Else?
- Apply download blocking and remove vulnerable versions from artifactory etc.
- Delete container images that are vulnerable to prevent accidental deployment
- Delete build artifacts that are vulnerable
- Make sure employees are not able to push to container registries and internal build artifact repository
- Clean maven/gradle caches from employee machines. In case there are privileged users who can update container registries or build artifacts from their machines
Uptycs also helps with Log4j remediation efforts, helping you quickly inventory vulnerable systems and prioritize patch efforts.