What is Shadow AI?

Overview of Shadow AI

Shadow AI is any use of AI in an organization that lacks outside approved controls, isn’t inventoried, and isn’t monitored for security. It could exist within your software supply chain or your cloud environment. What is shadow AI at its core? Some shadow AI examples include:

• Unsanctioned AI apps like chatbots, copilots, and summarizers
• Unapproved models/APIs like browser extensions, plug-ins, and public LLM endpoints
• AI-generated artifacts like code, scripts, and packages

Traditional AI often uses a centralized platform with governance, audit trails, and risk management to minimize security threats. Shadow AI is decentralized, has minimal controls in place, and lacks audit trails. Here’s a breakdown of some of the key differences between shadow AI and centralized AI:

Shadow AI is often a result of developers, analysts, and other teams using unauthorized AI for speed and convenience. Copying and pasting data into these AI tools and services can result in source code, trade secrets, and sensitive customer data being leaked. AI output also becomes images and packages, which creates a ripple effect that results in even more shadow AI artifacts.

Shadow IT Versus Shadow AI

Shadow IT is the use of any unauthorized hardware, software, or cloud services without the approval of an organization’s IT team. For example, team members within the organization might use unauthorized Software as a Service (SaaS), scripts, or unmanaged devices that can lead to security risks.

Because shadow IT has limited visibility and access control, it can pose a security threat to organizations. However, shadow AI adds unique risks to shadow IT because of the nature of AI tools and services. These risks include:

• Data leaving the organization via AI prompts
• AI hallucinations and insecure code
• Hidden third-party model providers and plug-ins
• Provenance gaps for AI-generated code and binaries
• AI assets associated with AI agents can cause disruptive operations

Shadow IT can also result in shadow AI. When you use unapproved SaaS with plug-ins or extensions enabled, AI features may be turned on by default.

What Are the Risks of Shadow AI?

Some of the common risks of shadow AI include:

• AI-generated code isn’t always secure, introducing common flaws like injection, authorization bypass, and server-side request forgery (SSRF) that create unnecessary security risks.
• Tokens pasted into prompts and leaked via logs/history can expose secrets.
• AI may suggest insecure dependencies.
• AI may delete or modify data that it shouldn’t.
• AI-built Dockerfiles and base images from unknown sources can pose a security threat.
• Build pipeline issues such as unreviewed code and AI scripts altering CI/CD steps without controls.
• Third-party AI may have access to sensitive internal systems.

There are also several data privacy concerns with shadow AI. Sensitive data like source code, configs, financials, and customer contracts can be exposed. You also don’t know where prompts go or how long they’re stored for, and third-party connectors may be able to exfiltrate data.

Additionally, shadow AI comes along with compliance and regulation liabilities. As an organization, you have to follow certain compliance and governance processes. With shadow AI, you don’t know who used what model, what data was input, or when. There may also be policy conflicts with common compliance frameworks, including SOC 2, ISO 27001, GDPR, HIPAA, PCI, and internal SDLC controls.

There are also potential IP and licensing risks due to unclear attribution and reuse rules and the potential incorporation of restricted materials. Plus, untracked artifacts and dependencies can enter builds and releases, which impacts the software supply chain.

Finally, all of these could result in irreversible damage to the reputation of organizations that experience any of these negative consequences of shadow AI.

Causes of Shadow AI

Shadow AI is always a result of unauthorized AI usage, but there are a few reasons why that occurs:

• User-driven innovation and convenience resulting in premature adoption of AI tools
• Lack of awareness and training leading to over-trust in outputs or failure to recognize the risk of sharing data through prompts
• No approved tool list, guardrails, or monitoring through AI with missing SDLC controls for AI-assisted code
• Organizational gaps like no inventory of packages, containers, build outputs, SBOMs, and model usage metadata can amplify shadow AI

Examples of Shadow AI

Looking at some examples of shadow AI can help you better understand how shadow AI can impact your organization. Here are some common scenarios in business environments where shadow AI is used:

• Developers using public LLMs to generate, refactor, or write Terraform/Kubernetes manifests and identify errors in logs
• Analysts using free AI tools to summarize internal documents
• Engineers installing browser extensions to capture form inputs
• Teams using unapproved “AI code review” bots to check their code
• Docker files containing unapproved third party AI models / AI API calls.

There are many different places where shadow AI can show up:

• With an OpenAI or Gemini key, shadow AI can be incorporated into virtually any software a developer is building, and gain access to any information the developer has
• Public chat assistants, “copilot” coding tools, and AI meeting summarizers
• Extensions and plug-ins within IDEs, browsers, and ticketing tools
• AI-powered code generators for IaC, pipelines, and container build files

In January 2023, Amazon noticed ChatGPT answers “closely matched” existing material from inside the company. As a result, Amazon reminded its employees not to share confidential information in ChatGPT prompts.

How to Manage Shadow AI Effectively

1. Detect and Measure Shadow AI Usage

The first step to managing shadow AI is detecting and measuring shadow AI usage. There are several ways to detect shadow AI usage:

• Network egress to common AI endpoints
• Browser extension inventories
• IAM logs and OAuth grants for AI plug-ins
• Full software supply chain scan
• Repo/CI patterns like AI-generated commit signatures and sudden dependency changes

You can also build an inventory of approved tools/models, which teams are using them, and which types of data can be used with those tools.

2. Build and Enforce AI Usage Policies

Create a checklist of AI usage policies that you can enforce. Include:

• A list of approved tools/models and procurement paths
• Data handling rules specifying what can/can’t be pasted (source code, secrets, customer data) and redaction requirements
• Logging and audit requirements
• Human review expectations for AI-generated code/config
• Third-party plug-in governance (connectors, permissions, retention)

Writing a policy is a necessary first step, and training workers on that policy is critical—but true protection requires uncompromising enforcement to ensure only approved AI assets are used. Crucially, this enforcement must be integrated in a way that is easy, frictionless, and simple for your teams to follow so they aren’t tempted to look for risky workarounds.

3. Integrate Shadow AI Into Your Security and Governance Frameworks

AI outputs must be treated as software supply chain inputs. Requirements for each AI output must include:

• Artifact provenance (which tool/model it came from, who generated it)
• Dependency and vulnerability scanning
• Policy gates before promotion
• SBOM generation and retention

You must also enforce DevSecOps policies in CI/CD and registries:

• Proactively block risky packages and images
• Quarantine unknown artifacts
• Promote only compliant builds

4. Provide Safe Alternatives So Teams Don’t Route Around IT

Providing a “secure AI” path is an excellent way to enhance AI SPM. Offer a path that includes enterprise accounts, single sign-on (SSO), logging, and retention controls.

Enable “secure-by-default” developer workflows like pre-approved templates for Dockerfiles/IaC, and use guardrails and automated checks to minimize friction for efficient workflows.

5. Incident Response and Continuous Improvement

When you discover shadow AI being used in your organization, there are three important steps to follow:

1. Contain: Revoke tokens, rotate secrets, and remove extensions
2. Assess: What data was shared and where it was stored
3. Remediate: Re-scan artifacts, rebuild clean, and update policies and training

By taking these steps, organizations transform unmanaged AI into governed assets, applying the same rigorous security and control checks used across the rest of their software supply chain. Important metrics to review include % AI usage on approved tools, number of blocked/quarantined artifacts, and time to remediate exposures.

Managing Shadow AI with JFrog

Putting a stop to shadow AI usage and identifying security threats is a multi-step process that requires time, effort, and the right tools. By utilizing Shadow AI Detection within JFrog AI Catalog, you gain one comprehensive view of all managed and unmanaged models across your platform.

JFrog AI Catalog uses Xray to scan your artifacts and repositories and automatically identifies unvetted models being used by your team. It labels models as Managed, Unmanaged, or Partially Managed, so you can easily see which assets need your attention. From there, you can approve safe models or block risky ones.

For more information, please visit our website, take a virtual tour, or set up a one-on-one demo at your convenience.