On March 24, 2026, the artificial intelligence development community experienced an unprecedented security catastrophe. LiteLLM, an essential open-source Python library used to route and manage API calls across hundreds of large language models, was weaponized in a highly sophisticated supply chain attack. Threat actors known as TeamPCP successfully published two malicious versions of the package (1.82.7 and 1.82.8) directly to the Python Package Index (PyPI).
With LiteLLM averaging 97 million monthly downloads and serving as a foundational dependency for industry titans like Stripe, Netflix, and Google alongside major AI frameworks such as CrewAI, DSPy, and MLflow, the magnitude of this compromise is staggering.
The malware deployed in these compromised versions was an aggressive information stealer and persistent backdoor. It was designed to systematically harvest SSH private keys, cloud provider credentials for AWS, GCP, and Azure, Kubernetes cluster configurations, CI/CD pipeline secrets, and even cryptocurrency wallets.
However, the LiteLLM compromise did not occur independently. It was the cascading result of a prior breach involving Aqua Security’s widely trusted Trivy vulnerability scanner. This blog post dives deep into how TeamPCP exploited a GitHub Actions misconfiguration, how that initial breach bridged into the LiteLLM build pipeline, the exact mechanics of the malicious payloads, and the steps organizations must take immediately to secure their environments.
Why LiteLLM?
To understand the gravity of this incident, one must understand the architectural role of LiteLLM in modern software development. As the generative AI ecosystem exploded, developers faced a fragmentation problem. OpenAI, Anthropic, Google Gemini, and countless open-source models hosted on platforms like HuggingFace all required different API integrations, distinct payload structures, and varied authentication methods.
LiteLLM solved this through the “Adapter” software pattern. It provided a single, unified “completion” function. A developer could write one block of code and simply change a string parameter from let’s say from “gpt-5” to “claude-4.5-opus” which instantly routed traffic to a different provider.
This simplicity made LiteLLM invaluable. It became the defacto gateway proxy for enterprise AI agents. If an organization was building AI software in Python, there was a high probability that LiteLLM was sitting in their dependency tree, either installed directly or pulled in transitively by orchestration frameworks. Consequently, any environment running LiteLLM inherently held high-privilege API keys, database connection strings, and cloud access tokens which makes it the ultimate holy grail for threat actors seeking to break into corporate infrastructure.
The Trivy Compromise and the Pwn Request
The root cause of the LiteLLM disaster began nearly a month prior, on February 27, 2026, targeting an entirely different entity: Aqua Security’s Trivy. Trivy is an open-source, industry-standard, all-in-one security scanner used to find vulnerabilities and misconfigurations in code repositories and container images. Ironically, the tool designed to secure CI/CD pipelines became the tool for their destruction.
The initial vector was a well-known, yet frequently overlooked, GitHub Actions misconfiguration involving the “pull_request_target” trigger.
In GitHub Actions, workflows can be triggered by various events. A standard “pull_request” event runs workflow code in the context of the user submitting the code. However, a “pull_request_target” event runs the workflow in the context of the base repository. This means it has access to the target repository’s high-privilege secrets and Personal Access Tokens (PATs). If a workflow using “pull_request_target” blindly checks out and executes untrusted code from a forked repository, an attacker can simply submit a pull request containing malicious instructions, which the CI/CD runner will then execute with administrative privileges.
On February 27, an autonomous AI bot operating under the handle “hackerbot-claw” (self-described as powered by Claude-Opus-4.5) exploited this exact vulnerability in the Trivy repository. The bot submitted Pull Request #10252. The PR was immediately closed, but the “API Diff Check” workflow had already been triggered. The runner executed the attacker’s payload, which exfiltrated a highly privileged Personal Access Token to an external server.
Armed with this token, the threat actors (later identified themselves as TeamPCP) seized total control of Aqua Security’s GitHub accound and by extension, the Trivy repository. Over the next several days, they engaged in catastrophic vandalism and supply chain poisoning. They made the repository private, deleted 178 legitimate GitHub releases, and published a malicious Visual Studio Code extension.
Most importantly, they targeted the “trivy-action” repository i.e. the official GitHub Action developers use to integrate Trivy into their own pipelines. Because Git tags are mutable (meaning a tag like “v0.34.2” can be force-pushed to point to a different, malicious commit hash), the attackers rewrote 75 of the 76 version tags for trivy-action.
To evade detection, they cloned the metadata of the original commits and copied the author names, timestamps, and commit messages before pointing the tags to their credential-stealing malware. For weeks, any developer relying on standard versions of the Trivy GitHub action was unknowingly executing compromised code in their build pipelines.
The CI/CD Bridge to LiteLLM
This brings the timeline to March 24, 2026. The maintainers of LiteLLM, acting as responsible open-source stewards, utilized security tools to ensure the integrity of their codebase. Their CI/CD pipeline included a script named “security_scans.sh” which ran Trivy.
However, the pipeline contained a fatal flaw: unpinned dependencies. The script executed a standard package manager command to install Trivy directly from its repository without specifying a strict, safe version hash.
As the Trivy distribution channels had been poisoned by TeamPCP, the LiteLLM CI/CD runner downloaded and executed the compromised Trivy binary. When the malicious Trivy scanner ran inside LiteLLM’s GitHub Actions environment, it operated with the full permissions of that runner.
The malware scanned the runner’s memory and environment variables, successfully locating and exfiltrating the “PYPI_PUBLISH_PASSWORD” – the highly guarded secret token used by LiteLLM’s maintainers to publish new releases to the official Python Package Index.
The bridge was complete. By compromising a security tool, the attackers had stolen the keys to one of the most widely used AI packages in the world.
Poisoning the Python Package Index
With the PyPI publishing token in hand, TeamPCP bypassed the LiteLLM GitHub repository entirely. There are no Git tags or commits in the official LiteLLM source code corresponding to the malware. Instead, the attackers built the malicious packages locally and pushed them directly to PyPI.
At 10:39 UTC on March 24, they published LiteLLM version 1.82.7. At 10:52 UTC, they escalated the attack by publishing version 1.82.8.
Because these packages were signed and uploaded using legitimate maintainer credentials, they passed all standard cryptographic and integrity checks utilized by package managers like pip. Any system, CI/CD pipeline, or developer running an unpinned installation command (e.g., pip install –upgrade litellm) immediately downloaded the malware.
READ MORE: How TeamPCP Weaponized the Global Security Supply Chain to Steal Half a Million Credentials
Deep Dive into the Malware Payload
The payloads deployed in versions 1.82.7 and 1.82.8 shows how deep TeamPCP went to obfuscate the infostealer and related code in the codebase. Security researchers from Snyk, Endor Labs, and SafeDep have reverse-engineered the packages, revealing a multi-stage execution chain.
Version 1.82.7: The Proxy Injection In the first malicious release, the attackers injected a double base64-encoded payload into the file “litellm/proxy/proxy_server.py”. This approach required the victim to explicitly import the proxy module in their Python code for the malware to execute. While damaging, it left a window where the package could be installed but remain dormant if that specific module was uncalled.
Version 1.82.8: The .pth Escalation and the Accidental Fork Bomb Just thirteen minutes later, the attackers released 1.82.8, utilizing a much more lethal execution trigger: the Python path configuration file (.pth).
The attackers included a file named “litellm_init.pth” in the site-packages directory. By design, Python’s site module extends system paths at interpreter startup by reading .pth files. Any line in a .pth file beginning with “import” is automatically passed to the exec() function. This means the malware did not require the developer to import LiteLLM at all. The payload executed the absolute millisecond the Python interpreter was spun up for any reason within that virtual environment.
Ironically, this aggressive execution method contained a critical implementation flaw that led to the attack’s rapid discovery. The payload utilized the “subprocess.Popen” command to run the data-stealing script in the background, ensuring it didn’t block the developer’s main application. However, because it spawned a new Python child process, that child process also triggered the .pth file, which spawned another child process, creating an infinite loop.
This resulted in an accidental “fork bomb.” Developers worldwide, including researchers at FutureSearch who first reported the anomaly, noticed their high-end workstations instantly running out of RAM, with CPUs pegged at 100%, and process viewers taking tens of seconds to load. The malware crashed the very systems it was trying to quietly infiltrate.
Data Exfiltration Mechanics Once running, the malware acted as an indiscriminate vacuum for sensitive data. The script systematically hunted for:
- Environment Variables: Capturing all loaded API keys, database URLs, and application secrets.
- Cloud Credentials: Scraping the default hidden directories for AWS, Google Cloud Platform (GCP), and Microsoft Azure configurations.
- SSH Keys: Harvesting all private keys and known_hosts files to enable lateral movement.
- CI/CD Secrets: Searching for Docker registry authentications and Kaniko credentials.
- Cryptocurrency Wallets: Specifically targeting local wallet files for Bitcoin, Ethereum, Solana, and Cardano.
The stolen data was immediately encrypted to evade network-level packet inspection. The malware generated a 32-byte AES-256 session key, encrypted the payload, and then encrypted that session key using a hardcoded 4096-bit RSA public key. The entire bundle was compressed into a file named “tpcp.tar.gz” and exfiltrated via HTTPS POST requests to a newly registered command-and-control domain: models.litellm.cloud.
If the malware detected that it was running inside a Kubernetes cluster (by locating specific service account tokens), it escalated from data theft to infrastructure takeover. It interacted with the Kubernetes API to attempt the deployment of privileged pods across every node in the cluster.
Furthermore, on standard Linux environments, it installed a persistent backdoor. It created a hidden Python script at “~/.config/sysmon/sysmon.py” and established a systemd user service (“sysmon.service”) to ensure the script ran on boot. This backdoor acted as a beacon, polling the attacker’s command-and-control server every five minutes to download and execute arbitrary secondary binaries.
Who is Affected?
The exposure window for versions 1.82.7 and 1.82.8 was approximately four hours before the Python Package Index security team permanently quarantined the package. Given LiteLLM’s download volume, this translates to hundreds of thousands of potential compromises.
- Direct Installs: Any developer who ran an unpinned install or upgrade command during the exposure window on March 24 is compromised.
- Transitive Dependencies: This is the most dangerous vector. Organizations that do not use LiteLLM directly, but rely on frameworks like CrewAI, Browser-Use, DSPy, Opik, or Mem0, may have pulled the malicious version automatically when those parent packages resolved their own dependencies.
- CI/CD Pipelines: Automated build pipelines that construct Docker images or run unit tests without strict lockfiles are the highest-risk targets. If a pipeline ran during the exposure window, the malware successfully executed, stealing the pipeline’s deployment credentials and effectively handing the attackers the keys to the organization’s production infrastructure.
- Cached Container Images: Docker images built during this window and pushed to registries will contain the malware. If deployed, these containers will initiate the data theft and persistence protocols inside the host environment.
Immediate Remediation and Incident Response Protocol
If your organization utilizes Python and AI frameworks, you must operate under the assumption of compromise until definitively proven otherwise. Downgrading the package is insufficient; the payload executes instantly and establishes persistence outside the Python environment.
1: Halt and Quarantine Immediately isolate any developer workstation, CI/CD runner, or production container that built or executed Python environments during the exposure window (roughly 09:00 to 13:30 UTC on March 24, 2026).
2: Check version of your installed packages. Execute a package listing command (e.g., pip show litellm). If the version returns as 1.82.7 or 1.82.8, the system is actively compromised.
3: Comoletelt delete the dev environment. Do not attempt to uninstall the package. You must entirely delete the virtual environment folder and recreate it from scratch.
4: Check for Persistence Artifacts. Scan file systems for the backdoor. Look for the directory structure “.config/sysmon/sysmon.py” in both user home directories and the root directory. Additionally, check the systemd service manager for any unrecognized services named “sysmon”.
5: If a machine is confirmed to have run the compromised versions, every single secret residing on that machine must be considered stolen and instantly revoked. This includes:
- Revoking and regenerating all AWS IAM keys, GCP service accounts, and Azure service principals.
- Invalidating all SSH private keys and removing corresponding public keys from authorized servers.
- Rotating all database passwords, OpenAI/Anthropic API keys, and third-party service tokens present in local .env files.
- Resetting all GitHub PATs and Docker registry credentials.
Lessons Learned and The Future of Open-Source Security
The LiteLLM supply chain attack represents a watershed moment in software security. It painfully illustrates that modern infrastructure is a house of cards built upon open-source dependencies, where a single point of failure can cascade into global compromise.
The primary failure was not in LiteLLM’s code, but in systemic dependency management. The incident underscores three non-negotiable practices for modern development:
First, the absolute necessity of Lockfiles. Organizations utilizing package managers that generate strict lockfiles (such as Poetry or uv) were completely immune to this attack, provided they had pinned LiteLLM to a known safe version prior to the incident. Bare installation commands without version constraints are a catastrophic risk in production environments.
Second, the danger of Mutable Tags. The Trivy compromise was only possible because GitHub tags can be overwritten. CI/CD pipelines must move away from referencing dynamic tags (like @v1) and instead pin GitHub Actions to immutable, cryptographically secure commit hashes.
Third, the principle of Least Privilege for Automation. The original Trivy breach succeeded because a Personal Access Token possessed privileges far exceeding what was necessary, allowing the deletion of releases and the publishing of code. CI/CD tokens must be strictly scoped, heavily monitored, and routinely rotated.
The cleanup from the LiteLLM and Trivy compromises will likely take months, as organizations untangle their dependency trees and hunt for deeply buried persistent backdoors. As the AI ecosystem continues its breakneck acceleration, this incident serves as a grim reminder: the tools we use to build the future are only as secure as the weakest link in their supply chain.
Frequently Asked Questions (FAQs)
Q: How do I know if my project was affected if I don’t use LiteLLM directly? A: You must inspect your transitive dependencies. Use dependency tree commands (like pipdeptree) to see if parent frameworks (such as CrewAI or DSPy) pulled LiteLLM into your environment. If you do not use strict lockfiles, you were likely exposed.
Q: Is it safe to use LiteLLM now? A: Yes. PyPI lifted the quarantine after permanently deleting the malicious 1.82.7 and 1.82.8 versions. Furthermore, the maintainers revoked all compromised publishing tokens. Installing the latest version (1.82.9 or higher) is currently deemed safe.
Q: Why didn’t antivirus software catch the malware? A: The payload was double base64-encoded, which successfully evaded many static analysis and signature-based scanning tools. Furthermore, executing standard Python libraries like “subprocess” is common behavior, masking the malicious intent from behavioral heuristics until it was too late.
Q: We use Trivy in our CI/CD. Are we compromised? A: If you ran Trivy via unpinned GitHub Actions (trivy-action) or downloaded the binary dynamically during the late February to mid-March window, your CI/CD pipeline secrets may have been exposed to TeamPCP. You should perform a comprehensive audit of your runner logs and rotate CI/CD secrets.
