Catching Log4j in the Wild: Find, Fix and Fortify

At many organizations, the surprise discovery that the widely used Log4Shell open source software has harbored a longtime critical vulnerability was as if Scrooge and the Grinch had teamed up for the biggest holiday heist of all. Incident response teams across the globe have scrambled to remediate thousands, if not millions of applications.

“For cybercriminals this is Christmas come early,” explained Theresa Payton, former White House CIO and current CEO of Fortalice Solutions. Attackers quickly exploited this vulnerability to install cryptocurrency mining malware, steal secure credentials, and more.

Responding to the crisis has meant weekends lost, parties missed, and travel plans canceled as teams try to find the needles across their many development haystacks.

Users of Artifactory and Xray, however, have had a lot of help. If you’re one of them, you can experience firsthand the long term benefits gained from making the JFrog DevOps Platform a central part of your mission-critical software development lifecycle (SDLC) process. 

With your accumulated packages, binaries, images, and metadata under Artifactory’s binary repository management, you already have everything you need to quickly remediate against the Apache log4j vulnerability – or any other zero-day security issue encountered.

We’ll explain how you can swiftly protect your organization using the JFrog Platform. You’ll learn how to:

• FIND – Use Artifactory’s build-info metadata and Xray’s deep-recursive scanning to discover every usage of the log4j package, including transitive dependencies, across your entire inventory of applications put into production.
• FIX – Once Xray has identified applications using a vulnerable log4j package, developers can update their source code to use the safer, updated version and produce a new build.
• FORTIFY – Use Xray to block any further builds using vulnerable versions of the log4j package, and any promotion to production release of log4j vulnerable builds already in your pipeline.

With your packages, binaries, images, and metadata under #JFrog #Artifactory and #Xray, you already have everything you need to quickly remediate against the Apache #log4j vulnerability. Click To Tweet

Find Every Use of Log4j in Your SBOMs

To produce a Software Bill of Materials (SBOM) for your applications, you’ve been publishing build-info metadata through the JFrog CLI or in your Pipelines CI/CD automation. This is one of the most essential best DevOps practices enabled by the JFrog Platform, making all of your builds fully traceable to all of their component sources.

“A big bonus for our developers is build metadata – when log4j hit, it was the easiest thing to generate a report of what apps had that vulnerable dependency, fix it, and we were good to go.”
– Jay Bieshaar, DevOps Engineer, Bendigo and Adelaide Bank

Armed with this data about your binaries, you can search all your builds for the vulnerable log4j package.With Artifactory’s rich search capabilities, you can also narrow your search to target the most immediately relevant builds, enabling you to focus your efforts and zero in on what needs your urgent attention.

For example, if you’ve been following the best DevOps practice of immutable build promotion, then you can be efficient by limiting your searches to only the Artifactory repositories that represent the final stage of your promotion chain (e.g., your “release” repo) for deployment to production environments – there’s no need to examine builds in Artifactory that failed tests or were blocked for other reasons.

Let’s look at the top ways you can use to find log4j or any vulnerable package:

Using the JFrog Platform Search Bar

The application search bar at the top of the JFrog Platform user interface provides an easy way to discover where any items are used across your repositories. Use the search bar in Artifactory to lookup the log4j packages that are cached in Artifactory remote repositories, as well as their locations in local repositories and builds.

Using Artifactory advanced search, you can filter your results as you need. For example, only versions of log4j 2.0 and greater have the vulnerability. Similarly, you will likely only need to search repositories holding production releases.

Using Artifactory Query Language

You can also use the find request of Artifactory Query Language (AQL) to search your repositories using complex criteria to produce a filtered list of builds.

For example, you can query for all the builds that use the log4j-core library:

builds.find({
  "$and" : [
  {"module.dependency.item.name":{"$match":"log4j-core"}}
  {"created": {"$gt": "2019-01-01"}},
  {"promotion.repo": {"$match":"maven-release"}}
  ]
}).include("name", "repo", "@version")

You can filter on a wide range of AQL field criteria as your needs dictate, such as:

• created – limit results to builds created only in the last year, or between certain dates.
• promotion.status – limit results to only those builds that have reached a given stage of the SDLC promotion pipeline (for example, “release”).
• promotion.repo – limit results to only those builds in a certain repository (for example, the repo holding builds released to production).

Using Xray CVE Search

As of December 10, the Apache Log4j security issue has been registered with the National Vulnerability Database (NVD) as CVE-2021-44228. That helps a software component analysis (SCA) tool like JFrog Xray to flag it as a recognized vulnerability. (Note: A newer CVE, CVE-2021-45105 has also been reported on the 2.16.0 update.)

Unlike most SCA tools, Xray gains extra power from its deep integration with Artifactory, enabling its deep-recursive scanning to discover every direct or transitive dependency on a vulnerable component in all of your Artifactory repositories.

Once Xray has updated its indexing, you can search for the latest CVE by its reference number from the Security & Compliance search bar.

This will provide you lists of where the CVE was found in the packages, builds, artifacts & release bundles you have stored.

Fix the Log4j Dependencies in Your Builds

Once you know which of your applications and builds are vulnerable, your development teams can update their source code to use the latest version of log4j deemed safe (currently v2.17.0), and produce new, remediated builds and release bundles. 

You can then deliver these more secure application releases to your various consumption points, such as through JFrog Distribution, or deploy them directly to your Kubernetes clusters through Helm charts.

If you can’t wait for developers, the priority resolution characteristics of Artifactory virtual repositories can be used to create interim, mitigating builds. This method is explained in our recent Log4shell technical blog post. 

Fortify Repositories Against Future Vulnerabilities

Once you have remediated all current applications, you’ll want to make sure log4j security issues – and any others that may yet be discovered – are kept out of any future releases or newly produced apps.

To prepare, you’ll need to make sure that the repositories you want Xray to scan (for example, your “release” repos) have been configured to enable indexing in Xray. Only these indexed repos will be available to Xray.

Use Xray policies and rules to automatically recognize threshold severity level vulnerabilities in your packages and builds in chosen repos. Setting Minimal Severity to High will include both the critical log4j issue, as well as the lesser vulnerabilities of recent updates. You can also configure automatic actions so that Xray can respond to a policy violation by sending a notification, triggering a webhook, or blocking download of the unsafe package.

You can associate these policies with Xray watches, and associate them with key repositories, builds, or bundles. If necessary, you can also assign the watch to all indexed repositories. In the watch, you can configure an Xray watch filter to limit the watch specifically to usage of Apache log4j.

Once the watch has been run, you can examine the violations Xray found, and all uses of vulnerable versions of Apache log4j will be included in your ongoing Xray security reports to help prevent future exposure.

You can even use JFrog integrations to send Xray alerts to PagerDuty or Xray reports to Slack, putting them immediately in front of incident response teams.

Remediate in Hours

As news outlets continue to report, incident response teams worldwide can expect to spend many days and even weeks fully mitigating their company’s broad exposure to the Apache log4j security issue.

But the over 8,000 customers of the JFrog Platform, whether in the cloud or on-prem, have the ready means to find, fix, and fortify their entire software supply chain from the log4j vulnerability in only hours. The best practices of DevSecOps through rich binaries management, SBOMs, and SCA enabled by Artifactory and Xray have helped many to save their weekends and holidays.

Xray’s scanning covers vulnerabilities in 18 different package formats. In addition to the Java libraries where Apache log4j resides, Xray can monitor repositories for JavaScript, PHP, Docker, Go, C++, and more.

Users of the #JFrog Platform have the ready means to find, fix, and fortify their entire software supply chain from the #log4j vulnerability in only hours. Click To Tweet

In a world where 92 percent of apps use open source code, this won’t be the last time a critical vulnerability is discovered in a widely used package. The log4j security issue follows fast on the heels of the Squirrel vulnerability, as well as last year’s SolarWinds hack. The World Economic Forum now ranks supply chain attacks among the top cybersecurity challenges.

For software development companies – which now means all companies – worrying about these incidents are something to lose sleep over. But they don’t have to be. Leaders of teams using the JFrog Platform can nestle all snug in their beds.

Let us show you visions of DevSecOps, with a personal demo of Xray.

What, you’re not using the JFrog Platform? It’s easy to start for free!