On the surface, some tools market DevSecOps capabilities as part of their software supply chain solution. Still, DevOps and Security teams who dig deeper into these tools will quickly spot some red flags, including:

• Packaging Competitor’s Open Source as an Enterprise solution: Selling a paid “security” solution that’s little more than a thin UI layer over someone else’s open-source tool.
• Built on Vulnerable Foundations: Relying on an outdated version of this open-source tool known to contain 24 publicly disclosed security vulnerabilities.
• Fails the Basics: Can’t detect well-known malicious packages.
• Noise Over Signal: Flags over 3,000 false positives from a single container image, flooding users with meaningless alerts.
• No Real Coverage: Lacking core DevSecOps capabilities like SAST, secrets detection, misconfiguration scanning, and runtime protection, leaving major blind spots.

Intro

As part of a research initiative at JFrog, which we also presented at InfoSecurity Europe 2025, we’ve benchmarked Application Security solutions across various stages of the software supply chain. The goal of this research is to uncover common blind spots in these solutions that could lead to the misidentification of critical security threats. As part of our research, we discovered cases in which companies with no security expertise have cobbled together a security solution based on OSS tools, promoting a false sense of security. This is most evident in the case of Cloudsmith.

In this blog post, we’ll present a case study on Cloudsmith’s solution for software supply chain security, analyzing its scanning capabilities and identifying potential blind spots related to application security. These findings are backed up by insights from our security experts team, so readers can take a closer look and draw their own conclusions.

How Cloudsmith Falls Short — And Why It Matters

Cloudsmith, following a similar approach to JFrog, provides an artifact management solution with a focus on binary-centric security. They promote themselves as a “fully managed solution to control, secure, and distribute everything that flows through your software supply chain.”

However, analysis indicates that they lack security expertise, do not provide a comprehensive software supply chain security solution, rely on open source software as their SCA solution, and have critical blind spots that can leave your software supply chain exposed.

Before delving into fine technical details regarding each of these points, here is a summary of our main findings –

> Cloudsmith’s security solution is simply a “wrapper” over an Open Source SCA Engine

Our research revealed strong indications that Cloudsmith uses Trivy under the hood as their SCA solution without stating this publicly. Customers are buying Cloudsmith but actually paying for and relying on an open-source free tool developed and maintained by a different company – Aqua Security, that sells a competing security solution

As shown in our findings, Cloudsmith’s apparent reliance on this open-source scanner, Trivy, as the foundation of its commercial product is a major red flag. This dependence is not only flawed, but can also expose Cloudsmith customers to multiple risks. If Trivy becomes unsupported or compromised, Cloudsmith’s entire SCA capability could suffer, leaving users vulnerable. Furthermore, any vulnerabilities in Trivy itself could immediately put both Cloudsmith and its clients at risk. In our tests, we found that they currently use a 3 month old version of Trivy, which is exposed to 24 known security vulnerabilities.

Open-source tools are generic by nature and lack the customization needed for advanced threat detection and mitigation strategies. By relying heavily on an open-source component maintained by another company, Cloudsmith is essentially offering a watered-down security solution that fails to meet standard software supply chain security capabilities, as demonstrated in this research.

> Cloudsmith has critical gaps in the implementation of their security features, leading to both false positives and false negatives

While Cloudsmith claims to identify malware and vulnerabilities in software packages, our research found significant security gaps:

• Malware Detection: Testing malware detection revealed Cloudsmith failed in the most basic case – reporting well-known malicious packages.
• SCA Capabilities:
  • High false positive rate: 3,171 false positives in a single image, generating excessive noise, wasting users’ time on chasing down irrelevant findings
  • Blind spots leading to false negatives:
    • By relying only on SBOM metadata, Cloudsmith failed to detect the main software component in the top 8 pulled Docker Hub containers, leading to 90+ missed CVEs, as these components are not installed via a package manager.
    • A lack of CVE detection accuracy, resulted in missing critical vulnerabilities in widely used components due to reliance on unenhanced public CVE information only.
  • Lack of CVE Prioritization: No prioritization by reachability or other risk metrics, causing an unmanageable volume of CVEs.
  • Lack of Extended CVE Info & Remediation: Offers only basic, public CVE data without actionable guidance.

> Cloudsmith provides the bare minimum in security features, lacking many of the features that are commonly found in software supply chain security today

Despite claiming they secure everything that flows through your software supply chain, we found Cloudsmith lacks robust support for:

• Shift-Left Approaches: No support for early security issue detection, such as source code scanning.
• Shift-Right Strategies: Absence of post-deployment security monitoring.

More specifically, the following features are not supported by Cloudsmith:

• SAST: Protection against zero-day vulnerabilities.
• Secrets Detection: Prevention of hardcoded tokens and passwords from getting exposed.
• Security Misconfigurations: Identification of improperly configured services, applications, libraries, and frameworks.
• Runtime Protection: Protection against live threats, including dynamic CVE prioritization.

Solving solely CVE issues can leave your supply chain vulnerable to attacks. Unlike Cloudsmith, the JFrog Platform’s security solutions comprehensively address every gap presented in this report. Let’s explore them in greater detail.

Comparing Security Features Apples to Apples – JFrog vs. Cloudsmith

Let’s dive into comparing features that exist in JFrog and are also present in Cloudsmith, according to each company’s documentation and APIs.

Malicious Package Detection

On their main page, Cloudsmith states that they identify malware in software packages:

Scanning excerpt from Cloudsmith’s documentation page

However, our testing reveals some critical gaps in their detection capabilities, even in the most basic cases.

During our testing we uploaded a Docker image containing a well-known malicious NPM package, multicogs:1.1.5.

Multicogs is a well-known malicious package, present in multiple advisory databases

 

Cloudsmith successfully identified the multicogs:1.1.5 package in its SBOM.

SBOM excerpt from Cloudsmith, identifying the multicogs package in a container

Despite the known malicious nature of this package, Cloudsmith failed to identify any associated malicious content. Below is a screenshot showing Cloudsmith’s detection of the security threats for the provided image, with no mention of malicious packages:

Cloudsmith screenshot, showing no malicious packages were detected

 

In contrast, JFrog accurately flagged multicogs:1.1.5 as a malicious package:

JFrog Xray screenshot, identifying the well-known malicious package “multicogs”

This is not an isolated case – uploading multiple malicious packages from multiple ecosystems resulted in zero malicious packages detected by Cloudsmith.

Identifying malicious packages is critical for safeguarding the software supply chain, especially considering attackers increasingly target developers, enticing them to download compromised packages.

Vulnerability Detection

Cloudsmith uses Trivy as their SCA engine

Due to reliance on an Open Source SCA Engine – you basically wind up paying for a freely available OSS package!

While analyzing the CycloneDX SBOM results from Cloudsmith, it becomes apparent that Cloudsmith’s SCA capabilities are entirely dependent on Trivy, a free open-source security scanner.

While Cloudsmith does not advertise this fact anywhere, it can be understood from scan results:

1. When exporting a CycloneDX SBOM file from Cloudsmith, we found the version 0.59.1 of Trivy listed in the `tools` section. This version, released in February, is outdated and susceptible to 24 well-known CVEs. This lack of timely updates not only diminishes Cloudsmith’s security coverage but also signals a reactive approach to maintaining security tools.

   

2. Cloudsmith and Trivy produce 100% identical vulnerability results when scanning 10 different Docker images, which in aggregate contain more than 1000 CVEs. It is very unlikely for two different SCA engines to produce 100% identical results on such a large dataset.
3. Cloudsmith reports some (non-CVE) vulnerabilities with the unique identifiers “TEMP” and “AQUA”, which are vulnerability identifiers used exclusively by Trivy.

Cloudsmith identifying a vulnerability with the Trivy-specific “AQUA” identifier

Problems in outsourcing your core logic to OSS components

While Trivy in itself is a very well regarded open-source scanner, it is problematic and misleading for a commercial company to base its core logic around such a component. This might be the reason why it is not presented anywhere on Cloudsmith’s website, documentation, or OSS projects. Essentially, Cloudsmith’s SCA performance is a direct reflection of Trivy’s detection capabilities, not an outcome of Cloudsmith’s own security innovations.

Depending solely on an open-source tool as a primary security solution introduces several critical risks:

• Lack of Control: If Trivy becomes unsupported, deprecated, or compromised, Cloudsmith’s entire SCA capability could collapse, leaving customers vulnerable.

• Security Gaps: Open-source tools are not immune to vulnerabilities. If a vulnerability is discovered in Trivy itself, Cloudsmith and its clients are immediately exposed. This is made even worse by not performing consistent updates, as can be seen in the outdated version currently in use by Cloudsmith.

• Missed Customization Opportunities: Open-source tools are designed for broad use cases, not tailored solutions. Relying solely on them, without adding extensive customization and upgrades, limits the ability to implement advanced, context-specific threat detection and mitigation strategies.

Ultimately, organizations seeking comprehensive and reliable security coverage should consider the implications of outsourcing critical security capabilities to open-source tools. At JFrog, we are committed to delivering robust, internally-developed tailored solutions powered by the research of our security experts group, to protect our customers and elevate security standards across the software supply chain.

CVE False Positives – When More is Not Better!

The effectiveness of a robust SCA solution is determined by its capacity to accurately identify software components and their corresponding vulnerabilities. This can be demonstrated by a low rate of false positives, as well as minimizing missed vulnerabilities, indicated by a low rate of false negatives.

Looking at the scan results of the same test image we used earlier for malicious package detection, we observed a significant discrepancy in the number of vulnerabilities identified by JFrog Xray and Cloudsmith:

• JFrog Xray: 195 unique vulnerabilities identified.
• Cloudsmith: 1,354 (786 unique) vulnerabilities identified.

Upon closer inspection, we discovered that 590 of the vulnerabilities reported by Cloudsmith were actually Linux kernel CVEs, which are irrelevant for Docker images since they do not have kernel. This is in direct contrast to Xray which identified 195 vulnerabilities that were all relevant for the Docker image. In this case, hundreds of false or irrelevant positive results could waste many hours of time and expend a huge cognitive load using Cloudsmith tools.

Linux Kernel CVEs appearing in Cloudsmith’s Docker test image scan results

This is not just a single isolated case, we observed it in 4 out of 8 public images analyzed, with the official Python Docker image showing a whopping 3,171 false positives caused by the same issue.

These findings underscore a risk in Cloudsmith’s scanning capabilities, resulting in an excessive number of false positives that create substantial noise in the vulnerability management process. When security and development teams are inundated with irrelevant findings, they risk wasting valuable time and resources chasing non-existent threats while potentially overlooking potentially damaging vulnerabilities.

Moreover, such inaccuracies can erode trust in the tool’s output and create the security version of “the boy who cried wolf,” where teams begin to question a tool’s credibility and wind up ignoring actual threats as a byproduct.

CVE Blindspots / False Negatives – Missing the point

Relying only on SBOM metadata creates SCA blind spots

Cloudsmith’s SCA solution does not employ advanced SBOM detection techniques. Like many naive SCA tools, Cloudsmith relies only on SBOM metadata obtained from package managers, embedded in binaries or other sources.

However, in reality, not all packages are installed through package managers or contain such metadata. Examples include:

• Compiling a package from source.
• Copying, transferring, or moving an entire package or part of it.
• Downloading packages using commands like wget or curl that do not go through a package manager.

This practice is especially relevant for Dockerfiles, where commands such as `COPY` and `ADD` can transfer files and directories containing binaries. More commonly, the `RUN` command is used with wget or curl to download specific package versions, which are then unarchived using commands like `tar`. This process bypasses traditional package manager workflows and can result in SCA tools misidentifying such components.

To assess how Cloudsmith handles these scenarios, we analyzed the official Apache HTTPD 2.4.49 Docker image.

The Dockerfile for Apache HTTPD shows that the main container application (Apache HTTPD) is downloaded using the wget command via RUN and then unarchived using the tar command:

Excerpts from httpd’s Dockerfile, showing httpd “installed” without a package manager

Comparing JFrog Xray and Cloudsmith results, we observed that only JFrog Xray detected the “main” component of this container – Apache HTTPD 2.4.49, and its associated vulnerabilities, while Cloudsmith completely missed it do its reliance only on naive techniques.

JFrog Xray scan of Apache HTTPD 2.4.49

In this example alone, there are 34 known CVEs that went unreported by Cloudsmith, including:

• 1 Applicable CVE

• 12 Critical-severity CVEs

• 17 High-severity CVEs

Again, this is not a unique case! In fact, analyzing 8 of the most pulled Docker images on Docker Hub revealed that in every single one, the main component of the docker image (e.g. The “node” component in the “node” docker image) or the main component’s runtime binaries were installed without a package manager. This leads to a significant number of critical false negatives of missed CVEs in Cloudsmith, as seen in the table below:

JFrog Xray is able to overcome these challenges and detect these vulnerabilities accurately by using multiple SCA techniques, such as:

1. Extraction of SBOM metadata from textual formats
2. Extraction of SBOM metadata from binary formats
3. Lookups in JFrog’s Hash-matching DB for binaries, powered by the JFrog Catalog
4. Analysis of unique features in binaries, such as Identifying version strings that are not part of the SBOM metadata

JFrog’s multiple SCA techniques result in more accurate component detection

CVE Detection Accuracy

In general, over 23,000 CVEs in JFrog Xray’s database have been verified by the security research team, either through vulnerability analysis, component verification, or confirmation of fixed versions. This proactive approach is crucial, as public data regarding CVEs can be incomplete, particularly in the initial days after a CVE is disclosed, and users are most vulnerable.

A notable example is the widely used component jackson-mapper-asl 1.9.13, which is vulnerable to over 50 CVEs. However, public sources, including Maven, only list 2 CVEs for this component. This discrepancy occurred since jackson-mapper-asl was renamed to jackson-databind after version 1.9.13, and unfortunately CVEs discovered later on jackson-databind were not “backported” as vulnerable on jackson-mapper-asl.

Maven page of jackson-mapper-asl showing only 2 relevant CVEs

Therefore, basic SCA scanners, including Cloudsmith, fail to detect the full list of CVEs when scanning the component. While Cloudsmith failed to scan the jackson-mapper-asl-1.9.13.jar file, it successfully scanned the jackson-mapper-asl-1.9.13.pom file and reports only these 2 CVEs:

Cloudsmith scan of jackson-mapper-asl 1.9.13, showing 2 vulnerabilities

In contrast, JFrog Xray successfully detected 56 vulnerabilities, including 25 with critical severity, effectively bridging the gap caused by incomplete public data.

JFrog Xray’s scan of jackson-mapper-asl 1.9.13 showing 56 vulnerabilities (Click to enlarge)

Within the false negative results, we can see CVE-2018-19360 which should be applicable to “jackson-mapper-asl” as well as “jackson-databind”.This can readily be seen by the successful exploitation of this CVE on jackson-mapper-asl by the JFrog Security Research team as seen below:

Successful exploitation of CVE-2018-19360 on jackson-mapper-asl 1.9.13 using marshalsec

Lack of CVE Exploitability Analysis

When scanning for vulnerabilities, the sheer volume of detected CVEs can be overwhelming, making effective CVE prioritization a critical capability for security solutions. Robust security tools should incorporate exploitability analysis for detected vulnerabilities, with the highest priority given to CVEs that are identified as exploitable within the application.

According to our findings, Cloudsmith lacks a CVE exploitability analysis. It does not provide information on exploitability, nor in-depth severity assessments and only provides public severity scores, whose limitations will be discussed next.

To illustrate the impact of CVE exploitability analysis, we used Cloudsmith and JFrog Xray to scan 8 of the most downloaded Docker Hub containers (using versions from a year ago) and compared the results:

Comparing total vs. applicable CVEs in the top 8 Docker Hub containers

JFrog Xray uncovered 1,398 unique CVEs, while Cloudsmith reported more than 12k CVEs. It’s unreasonable to expect security or development teams to handle such a large number of vulnerabilities. Without prioritization, this overwhelming number becomes a bottleneck for security teams.

Note that the huge amount of detected CVEs by Cloudsmith is again due to false positives by incorrectly including kernel CVEs.

However, by focusing on exploitability (applicability), as illustrated in the table above, only 31 CVEs (2%!) were identified as applicable by JFrog Contextual Analysis, a drastic reduction that enables more targeted and effective remediation.

CVE Applicability Example

One of the reported CVEs for the MariaDB Docker image (from the table above) is CVE-2024-12084, affecting the rsync daemon. This CVE is only applicable when SHA support is compiled into the rsync binary. This can be determined by checking if any of the following strings are present in the binary:

• sha1
• sha256
• sha512

According to JFrog’s Contextual Analysis, these strings were found in the rsync binary in the mariadb:11.2.3 container, making the CVE applicable.

JFrog’s contextual analysis evidence for CVE-2024-12084 (Click to enlarge)

Non-Applicable Example

One of the reported CVEs for the Apache HTTPD Docker image is CVE-2024-38474, affecting the Apache HTTPD daemon. CVE-2024-38474 is applicable only in a non-default configuration of Apache, specifically when the `RewriteRule` directive uses external input in its substitution string in the configuration files `httpd.conf` and `.htaccess`.

Another potential applicability check involves rules containing a flag that directs the path to a handler (e.g. H=cgi-script).

According to JFrog Xray, none of these preconditions were detected, making the CVE non-applicable:

JFrog’s contextual analysis evidence for CVE-2024-38474 (Click to enlarge)

This reduction, from thousands of CVEs to a focused list of 31, is not unique to the above images. According to JFrog Xray’s anonymous usage statistics coming from our SaaS customers, on average, 85% of all high and critical severity CVEs were flagged as not exploitable (applicable) by Contextual Analysis. This statistic is not just a numeric decrease but a strategic shift. It underscores the critical importance of CVE exploitability analysis in transforming vulnerability management from a daunting backlog to a focused, actionable plan.

JFrog Advanced Security currently recognizes 1900+ CVEs with Contextual Analysis, providing an average coverage rate of 80% of all high and critical severity CVEs in scans done by Xray cloud customers. This takes into account not just Reachability, whether the vulnerable function is called by the provided source code or binary files, but also considers:

• Binary Symbols/Strings – Whether the vulnerable binary is compiled with the vulnerable functionality.
• Configuration parsing – Whether the textual or binary configuration of the vulnerable component allows the vulnerability to be exploited.
• Command line flags – Whether the vulnerable component is executed with command-line flags that enable the vulnerable functionality.
• Runtime environments – The OS in use, CPU Architecture, Linux Distribution, CPU Bitness and permissions of the vulnerable binaries

Relying only on public severity ratings

Alongside proper exploitability analysis, accurate vulnerability severity ratings are essential as well. When you have to rely only on public sources to determine CVE severity, without the additional analysis of security research professionals, then you don’t have the ability to distinguish critical and non-critical or completely irrelevant vulnerabilities.

JFrog offers its own severity ratings based on comprehensive research conducted by the JFrog Security Research team. This information includes specific severity reasons, considering factors such as exploit prerequisites and actual impact in real-world scenarios.

As seen in the screenshot below, after researching CVE-2023-29827, the JFrog Security Research team opted to downgrade the severity from Critical to Low, as the prerequisites for exploitation are extremely unlikely to exist in any real-world scenario – which even prompted the vendor to dispute the CVE.

Reducing CVE-2023-29827 from Critical to Low severity with full explanations (Click to enlarge)

Proper severity ratings prevent unnecessary panic and help organizations focus on genuine risks. These factors are often not taken into consideration in the traditional public CVSS scores.

In the tests we performed, the scan results generated by Cloudsmith clearly lack tailored severity ratings and rely solely on public data sources, potentially leading to deficient risk assessment. Moreover, even important public CVE data sources that can help prioritize vulnerabilities are missing:

• CISA KEV (Known Exploited Vulnerabilities): Identifying CVEs actively exploited in the wild.

• FIRST EPSS (Exploit Prediction Scoring System): Highlighting vulnerabilities likely to be exploited in the near future.

• Vendor Advisories and Threat Feeds: Monitoring vendor alerts and security bulletins for emerging threats.

JFrog Catalog provides KEV and EPSS data for further prioritization (Click to enlarge)

Lastly, with NVD experiencing delays in enriching CVE data, including CVSS, CPE, and CWE, relying on NVD as a single source of truth can limit the effectiveness of vulnerability management, leaving organizations with insufficient information to make well-informed decisions.

Lack of extended CVE information & remediation

A significant gap in Cloudsmith’s vulnerability management approach is the lack of extended CVE information and actionable remediation guidance. Unlike JFrog, which maintains an in-house security research team, Cloudsmith’s CVE information is limited to basic public data, lacking deeper insights that can significantly impact risk assessment and remediation efforts.

When managing thousands of vulnerabilities, it’s not just about knowing which CVEs are present, it’s about understanding their real-world impact and how your developers can fix them effectively.

In comparison, JFrog has 1500+ CVEs with extended research and suggested remediation information and a custom severity assigned. About 900+ of these CVEs have a different severity than their official severity.

While the CVE information provided by Cloudsmith is basic:

JFrog Xray presents more detailed and actionable information:

Example of a CVE with custom remediation steps (Click to enlarge)

Example of a CVE with extended information (Click to enlarge)

Critical Security Features Not Covered By Cloudsmith

So far, we’ve focused on assessing Cloudsmith’s SCA capabilities, but software supply chain security is more than just SCA. While SCA is useful for identifying known vulnerabilities, there are many other security issues within the software supply chain that Cloudsmith does not cover. Their claim to “secure everything that flows through your software supply chain” is completely misleading, as they only provide SCA. Here are some of the most important software supply chain security features that are missing in Cloudsmith’s solution:

SAST (Static Application Security Testing)

SAST plays a pivotal role in application security by enabling developers to identify vulnerabilities within their own code, commonly referred to as first-party code, prior to deployment. Given that Software Composition Analysis (SCA) is only effective in detecting known vulnerabilities, implementing a SAST code scanning solution is essential to reduce the risk of exploitation.

SQL injection detected by JFrog SAST (Click to enlarge)

Secrets Detection

Detecting secrets is also a critical component of application security since it uncovers sensitive data like passwords, API keys, and tokens hidden in the code before it goes live. Although some static analysis tools can detect vulnerabilities, they often miss these embedded secrets. In some extreme cases, a leaked secret can even compromise an entire software ecosystem.

A generic secret leak detected by JFrog Secret Detection (Click to enlarge)

Security Misconfigurations

Security misconfigurations are another important threat that should be considered in software supply chain security, because improperly set up services, applications, libraries, and frameworks can expose significant vulnerabilities. Even well-designed systems with no CVEs can be compromised due to overlooked settings or insecure default configurations.

NGINX misconfigured CORS policy detected by JFrog Advanced Security (Click to enlarge)

Runtime Protection

Runtime protection is crucial for identifying where vulnerable code operates in a live environment, allowing for the detection of active threats. Connecting these runtime findings with insights from Software Composition Analysis (SCA) provides a clearer understanding of the threat landscape, enabling the prioritization of critical issues for resolution.

The absence of a runtime security solution complicates the prioritization of vulnerabilities, restricts the ability to identify and manage vulnerable running artifacts, and creates a significant blind spot in software supply chain security.

JFrog Runtime – CVE applicability insights based on live assessment (Click to enlarge)

Conclusion

Securing the software supply chain requires a comprehensive strategy that goes beyond basic vulnerability scanning. Cloudsmith’s reliance solely on Trivy, a free open-source tool, without a dedicated security research team to analyze vulnerabilities, means they cannot adapt to evolving threats, and assess real-world exploitability, leaving software supply chains exposed to critical security threats. When you use Cloudsmith, you are essentially paying for someone to run an OSS tool in the cloud.

One concern raised by our team is the notable volume of false positives observed, which is wasteful for users due to the high signal-to-noise ratio in their security workflows – while still managing to report critical false negatives, which may leave customer applications exposed to real vulnerabilities.

They also lack critical software supply chain security features, including the absence of SAST, Runtime Protection, Secret Detection, Malicious Package Detection and providing CVE Prioritization approaches with CVE exploitability analysis.

In contrast, JFrog addresses these gaps by integrating advanced SCA and SAST, robust CVE applicability analysis and runtime protection, providing more accurate vulnerability findings that are also actionable. By leveraging proprietary threat intelligence and maintaining in-house research expertise, JFrog not only mitigates risks associated with open-source dependencies but also offers a more complete security solution that enables organizations to protect against emerging threats effectively.

Ultimately, a binary-centric security approach that includes source code and binary analysis, runtime monitoring, and enriched vulnerability data is essential for comprehensive software supply chain protection – it is here that JFrog clearly distinguishes itself from Cloudsmith.

To see for yourself how JFrog can provide comprehensive cloud-native and on-prem application security for your organization, take an online tour, schedule a one-on-one demo or start a free trial at your convenience.