The Discovery Debt: Why 2,500 Companies Paid in Months for Visibility They Deferred

In March 2026, a threat actor group called TeamPCP orchestrated the largest AI supply chain attack on record. Over 2,500 companies discovered that the LiteLLM package, a widely used AI framework, had been poisoned. What made this attack different from typical software breaches was not the target, but the path.

CloudSEK’s research confirmed the scale: 2,500+ organizations and 434,000 CI/CD pipelines. The FBI later warned that affiliated actors would likely weaponize the harvested credentials long after the initial intrusion, meaning supply chain attacks remain a real possibility as a follow-up campaign.

The compromise did not start with LiteLLM directly. Instead, attackers leaked an automation token that gave them access to Trivy, a trusted security scanner. LiteLLM’s CI/CD build process then pulled an unpinned copy of Trivy, meaning it automatically installed whatever was latest. When a poisoned Trivy version flowed into the LiteLLM release cycle, it shaped the code that went to PyPI. From there, 434,000 individual CI/CD pipelines across 2,500 organizations automatically downloaded and executed the compromised package, and the entire cascade took 40 minutes.

What matters for IT leaders is this: 2,500 companies did not know LiteLLM was critical to their infrastructure until the breach forced the question. None of them had mapped which build pipelines pulled dependencies from which upstream sources. None of them knew, in advance, what credentials sat on each pipeline and what downstream systems those credentials could access.

That gap between what a supply chain actually looks like and what an organization can see is where modern breaches succeed.

The Cascade: One Token to 434,000 Pipelines

To understand why the LiteLLM breach reached such scale, the architecture matters. Here is how the chain unfolded.

The attack sequence:

1. A token that authenticated LiteLLM’s CI/CD system to Trivy’s repository was leaked. The token had been rotated as part of normal credential hygiene, but not fully revoked. This left a window, approximately 20 days, during which an attacker could force-push poisoned code to Trivy’s repository under the tag of legitimate versions. Downstream systems that pulled Trivy unpinned from the package manager still saw what looked like authentic, signed releases.
2. LiteLLM’s own build process consumed Trivy from the system package manager without locking it to a specific version. This is common practice. Unpinned dependencies pull security patches automatically. In isolated systems, this is good hygiene. In a supply chain context, unpinned trust becomes a liability. The poisoned Trivy created a poisoned LiteLLM build.
3. LiteLLM published versions 1.82.7 and 1.82.8 to PyPI, Python’s central package repository, with malicious code embedded in a .pth file. This distinction is important here. A .pth file runs at Python interpreter startup, meaning it executes whenever Python is loaded, not just when LiteLLM is explicitly imported. This bypassed the common practice of running pip install with the \–ignore-scripts flag, which teams use to prevent arbitrary code execution during installations.
4. Automated CI/CD systems across 2,500 companies ran their scheduled builds. Because developers had written requirements like “install litellm” without specifying a version, the dependency resolver pulled the latest package from PyPI. In minutes, 434,000 pipelines had downloaded and executed the poisoned code.
5. On each pipeline, the malicious code escalated to root and harvested everything visible to the process:
   • AWS credentials from instance metadata services
   • Kubernetes tokens from mounted paths
   • GitHub tokens from environment variables
   • SSH keys from disk
   • CI/CD secrets that GitHub ordinarily masks from logs (scraped directly from process memory)
   • API keys for internal services and cloud platforms

The loot was encrypted with an attacker-controlled RSA key and transmitted to a typosquatted domain. In cases where exfiltration failed, the malware created a public repository inside the victim’s own GitHub account and uploaded stolen credentials there as a release asset.

In 40 minutes, compromised pipelines accessed their local secrets and the entire privilege stack of every company that used them.

The Visibility Problem

Here is the structural lesson: none of these 2,500 companies had answered five basic questions about their supply chains before the breach.

The five questions:

1. Do we actually use LiteLLM? Companies discovered the answer after incident response began, not before. Many found themselves running LiteLLM in pipelines they had forgotten about, or in dependencies several layers deep that nobody had documented.
2. If LiteLLM is compromised, which downstream systems do we lose? This requires mapping transitive dependencies. Not just “we use LiteLLM,” but “we use LiteLLM, which calls ChatGPT, which returns data to our internal knowledge base, which feeds our customer-facing app.” Breaking any link in that chain is business impact.
3. What credentials sit on the pipelines that run LiteLLM? Most companies could not answer this without audit logs. In practice, they assumed the worst: all credentials, everywhere. This drove the broad rotation response that followed, an approach that works but is both expensive and incomplete because it assumes you have found all credentials, which most organizations have not.
4. Can we see which pipelines pull this package and when? Without scheduled, high-frequency discovery and dependency scanning, the answer is no. Log files capture execution history, not the current state. An audit weeks after the breach shows you what ran, not what is vulnerable now.
5. If we need to isolate this package immediately, which build systems can we safely shut down without cascading service loss? Without service mapping showing which applications depend on which pipelines, this becomes educated guessing at best.

Answering these questions requires visibility into three layers: the discovered state of your infrastructure, the dependency relationships that connect services and components, and the ownership and isolation boundaries that let you predict change impact through tools like ViVID™ Service Mapping.

This visibility layer shapes how fast you respond when a breach is detected, and it determines whether you navigate crisis with clarity or chaos.

The Automation Amplifier

Here is why the 40-minute window mattered.

If 2,500 companies had to manually pull and test each dependency update, the breach would have reached maybe 10 organizations before removal. The remaining 2,490 would have discovered it through incident reports and applied patches during normal maintenance windows.

Instead, automation compressed an attack that could have taken months into one that completed in minutes. CI/CD systems do not ask permission. They pull dependencies as scheduled, test them, and deploy them automatically, at machine speed. The faster you deploy, the wider the attack surface during the poisoned package’s brief window of exposure.

This is the cost of modern development velocity. Speed of deployment is worthwhile for organizations. It lets teams move fast and innovate. But speed without visibility into what you are pulling and where it is landing creates risk. It means a poisoned dependency can infect hundreds of organizations before humans even notice something is wrong.

The industry response has been check your packages more carefully and “*update faster when patches are published*.” Both approaches are reasonable, but they miss the deeper point. The LiteLLM attack succeeded because the organization using it could not see that the package was critical until the attack forced the question.

The Structural Cost of Discovery Timing

The 2,500 companies that survived the LiteLLM breach with minimal impact had already invested in supply chain visibility before the attack hit. They could answer the five questions instantly at 3 a.m. when the breach was detected. They could say “LiteLLM runs in these three pipelines, with these credentials, under these teams,” and narrow the blast radius immediately.

The companies without this visibility faced a harder version of the same question: do we even use this package? Without logs showing who pulled LiteLLM and without a CMDB reflecting current infrastructure, that question could not be answered quickly. CloudSEK’s own incident response guidance lays out how much work follows from that gap. Credential rotation and runner isolation within the first 24 hours, environment rebuilding and audit log review within 72 hours, and dependency pinning with continuous monitoring as an ongoing control after that. Reaching that timeline still requires knowing what to isolate first, which is exactly the scope that discovery has to establish before remediation can begin.

Here is the asymmetry: continuous visibility costs resources predictably. Discovering your infrastructure during a breach costs resources unpredictably and cascades across your organization. Incident response during a supply chain attack requires both speed and accuracy at a time when you have neither.

This matters acutely for AI infrastructure. LiteLLM is a control point between applications and AI models. An attacker who compromises that layer reaches every system the pipeline powers. Shadow AI, unmanaged models, and unauthorized agent deployments remain invisible to traditional discovery. Yet they inherit the same privilege as the infrastructure around them, making them strategic targets for exactly this reason.

The choice is not whether the next supply chain attack will test your visibility. Rather, it is whether you will have already built it on your timeline or be forced to build it during a crisis.

The Visibility Question

The 2,500 companies in the LiteLLM breach discovered a simple truth: you cannot manage what you cannot see. They saw it during incident response, the hardest and most expensive time to learn. The real test is whether your organization will see it proactively, before crisis forces the choice.

Three components make that possible. IT Discovery builds a frequently updated inventory of what is actually running, refreshed on a schedule instead of reconstructed once a year. A CMDB organizes that inventory into configuration items with ownership, criticality, and relationships attached, so it reflects today’s infrastructure rather than last quarter’s. ViVID™ Service Mapping connects those configuration items into the services the business actually depends on, showing blast radius before a change goes live or an incident forces the question. Together, they turn the five questions from the LiteLLM breach into ones your team can answer before a breach, not during one.

Supply chain attacks will continue to evolve; the vectors will change, the targets will shift, but the one constant is that they will be faster than human response unless you have built visibility first.

The companies building this foundation now are spending resources predictably. The companies that defer it are betting that the next supply chain attack will not affect them. That bet has a cost too, and it is just invisible until the bet fails.

See how Trusted Runtime Truth turns discovery-sourced inventory into explainable blast radius before the next package compromise forces the question.

Schedule a demo to see how Virima brings discovery, CMDB, and service mapping together into the visibility supply chain security requires.

Frequently Asked Questions