Detect Shadow AI

Identify and control unmanaged models across your platform

Introduction

Uncontrolled use of AI models introduces security, compliance, and cost risks.

JFrog’s Shadow AI detection helps identify and bring unmanaged AI models into the AI Catalog, enabling you to apply governance and security policies.

By scanning all artifacts across the JFrog Platform, we identify what models are being used, keeping you up to date with what is in your system.

Shadow AI detection gives you the ability to manage and prevent the entry of unvetted ai assets into your system.

Note

Shadow AI Detection is available to all organizations with an active AI Catalog subscription

How does the Shadow AI Detection process work?

The Detection tab provides a single view of all AI models discovered in your JFrog Platform, whether managed or unmanaged. To detect models, your JFrog system uses Xray to scan artifacts.

According to your Xray scan settings, repositories are scanned to detect models.

If no models appear in the Detection tab, verify that the correct repositories are being scanned by Xray.

Note

For Administrators only: To verify or select repositories to scan:

In the Administration Model, navigate to Xray Settings > Indexed Resources.
Browse the list of repositories displayed. If the repositories you want to be scanned are not selected, click Add a Repository.
Select the repositories you want Xray to scan to detect models and the arrow button to move it into the Selected Repositoriescolumn.
Click Save.

Currently, Shadow AI detection supports only model packages.

In the Detection tab, models are automatically grouped into four categories to help you prioritize review and governance actions:

Category	Description
Managed	Every instance of the model is either allowed in the ai catalog or blocked by security policies. No instances of the model are unmanaged.
Unmanaged	No instances of the model are allowed in ai catalog or blocked by security policies.
Partially Managed	Models approved for use in at least one project and/or blocked in at least one project, but not yet fully governed across other instances.

For all categories, the system indicates if the model is 'malicious', meaning xray flagged ia posing a high security risk.

A table of detected models is displayed showing:

Model name, provider, and type
Governance status (Managed, Unmanaged, Partially Managed
No. of artifacts in which the model was detected

You can click on a model to open the model pane on the right, where you can see in which artifacts and repositories the model can be found and status of the model for all projects.

How to Allow Unmanaged or Partially Managed Models

Allowing a model will bring an unmanaged or partially managed model under governance in the AI catalog.

To allow a detected model:

Either:
1. In the Detections tab, click the Manage button adjacent to the unmanaged model you want to govern (in the Actions column.
  OR
2. Click anywhere on the model row, and then in the model pane, click Manage.
Select the projects for which you wish to allow the model, and click Allow Selected.
Click Done to close the window.

The model is added to the AI Catalog and becomes Managed for that project and appears in the Registry page.

Once all instances of a model are allowed, its overall status updates to Managed. If a model still has unmanaged instances, then it’s status is indicated as Partially Managed.

Keep your System Secure

Shadow IAI enables you to detect all AI assets in your system. You may need to block specific AI assets due for security and compliance reasons. To do this you need to block the assets via security policies.

How does Shadow AI Integrate with JFrog Security?

Shadow AI Detection integrates tightly with JFrog’s security components:

Detection:
- JFrog Xray: Provides detection data, model-to-artifact mapping, and malicious flags.
- JFrog Advanced Security (JAS): Identifies external API calls through source-code analysis.
Block Policies:
- For local repositories: Blocking applies Xray's Download Block policy.
- For remote repositories: Create a Curation by label policy that blocks the model from the cache in remote repositories.
Both these policies, block future downloads of the blocked model for that rep ository.
Note:
Note
In the AI Catalog, models detected by Xray display an indicator showing “Detected in x artifacts” along with the associated risk level.

How to Block Models

To block a model:

In the AI Catalog, open the Discovery tab.
Select (click) the model to be blocked.
Click the ... button In the top right corner, and click Block.
Select the project for which the model is blocked, and click Block.

See Setting up Xray to index and scan your repositories