One key nobody revoked, four months later a malicious PyPI package

Summary

On October 21, 2026, LiteLLM's release pipeline published a malicious litellm==1.107.0 to PyPI, Docker Hub and GHCR. The package was signed and shipped through the official process, so every signature check passed.

The root cause was not a software vulnerability. It was an administrative key committed to a public repo on May 27, 2026 and never revoked. More than four months later, an attacker used that key to obtain an org owner's GitHub CLI token, then kept going until they could sign a release.

The most dangerous property of the payload: the bundled files execute automatically when Python starts, even if the application never does import litellm.

Priority actions:

• If you installed litellm==1.107.0 or pulled v1.107.0, latest, main-latest or main-stable on October 21: treat every credential on that host as exposed, and note that the startup files live outside the application directory, so removing the package alone does not clean it.
• Audit every key that was once committed by mistake and then "noted for revocation". This incident exists because exactly one follow-up task went undone.

What the leaked file looked like

The file was harnesses/opencode/opencode.json in the public repo LiteLLM-Labs/lite-harness, a configuration file for an agent harness.

Four things stand out from its structure alone, without knowing any key value:

1. The same sk-… key was hardcoded in four separate places in one file: in the Authorization: Bearer header of three remote MCP servers pointing at an internal gateway, and in the provider's apiKey field. One key serving several purposes means revoking it breaks several things at once, and that is exactly the kind of friction that makes people postpone revocation.
2. An API key for a third-party sandbox service sat in an environment variable.
3. Placeholders such as sk-dev-master-key-change-me for a LAP endpoint on localhost. In other words, this was a development config with sample values and real values mixed together.
4. An absolute path on a developer's machine was exposed in the command section. Nothing about the file is sophisticated. It is what happens every day in every organization: a dev config committed for convenience, with one real value hidden among the samples.

Four silent months

According to LiteLLM's RCA, the mistake was caught quickly. Roughly three hours later a follow-up commit removed the file, and the exposed keys were noted for revocation.

The revocation never happened. LiteLLM states it plainly as human error: "nobody followed through on revoking the administrative key".

One more detail matters: the original commit remained in the repo's public history until October 22, a day after the incident broke. Deleting a file in a later commit does not remove its contents from Git history.

Source note: The Hacker News reports that the commit was "force-pushed over". LiteLLM's official RCA says otherwise: a follow-up commit removed the file, and the original commit stayed in public history. This post follows the RCA.

The key remained valid for more than four months.

The seven-step chain

All timestamps below are UTC, taken from the detailed timeline in the RCA.

Step 1 — An old key traded for a GitHub token (Oct 10, 21:13–21:28)

From IP 69.48.229.143, the attacker called the LiteLLM Agent Platform API using the mistakenly committed administrative key. The platform was an internal experimental project that let self-hosted cloud agents open pull requests on GitHub.

At 21:27:10, a GET /api/v1/managed_agents/agents?limit=100 request returned the configuration of every managed agent. One of them held a GitHub CLI OAuth token belonging to one of the two org owners.

The pivotal detail: what kind of token was stolen

This is the part worth dwelling on longest.

The token configured in the Agent Platform was not a Personal Access Token scoped to specific repositories and capabilities. It was a GitHub CLI OAuth token tied to an org owner account. As LiteLLM describes it, this type of token is:

• Unscoped
• Behind 2FA, meaning two-factor authentication has already been cleared
• Not subject to GitHub's "super-user" re-auth flow for sensitive organization- or repository-level operations From this point on, the attacker had owner-level access to the entire GitHub organization.

Steps 2–3 — The victim's own CI becomes the exfiltration channel (Oct 10 and Oct 12)

The attacker needed no new tooling. They used GitHub Actions.

The pattern, described in detail by Socket:

1. Create a throwaway branch named ci-tmp-<hex> from a recent legitimate commit so the push looks unremarkable
2. Push a commit adding .github/workflows/cache.yml, with an equally unremarkable subject: "ci: bump cache config"
3. The workflow is configured on: [push], so it runs immediately
4. Delete the branch Results:

Time	Branch	Repo	Obtained
22:11:45	ci-tmp-a351cc	BerriAI/litellm	6 secrets, including GH_TOKEN, the GitHub CLI token of a second maintainer
22:12:09	ci-tmp-be33e9	BerriAI/litellm-auto	RELEASE_PLEASE_TOKEN, the GitHub CLI token of a third maintainer

Both run records were deleted within three minutes, at 22:14:14 and 22:14:31.

On October 12, a second wave expanded into several other private repositories. According to Socket's analysis, the harvest covered GitHub tokens and API keys for Linear, a LiteLLM deployment, Together AI, plus a GitHub App's identity and signing key; the later wave added keys for Anthropic, OpenAI, AWS Bedrock, Tavily, a GitGuardian secret-scanning tenant, along with a Playwright storage state, PostgreSQL connection details and a password-bearing Redis endpoint.

The data went to two different webhook[.]site endpoints.

Step 4 — Repointing the release pipeline at a fork (Oct 21, 10:08–11:08)

The release workflows live in the private repo BerriAI/project-releaser, and they hardcoded BerriAI as the source owner, so no input allowed building from a fork. To build from one, the attacker had to modify the workflow itself.

They authored commit 9cf97e9 ("ci: release from a fork source repository"), adding a source_owner input that defaults to BerriAI but accepts any organization.

Then came the five most instructive seconds of the whole incident:

10:08:29   Push the commit with the second maintainer's token
           -> BLOCKED by branch protection
              (pull request required, 2 passing checks, codeowner approval)
 
10:08:34   Push again with the org owner's token
           -> SUCCEEDED, branch protection overridden

At 10:46:39, the staging workflow was dispatched with its source pointing at Berri-AI/litellm, a fork inside the lookalike organization Berri-AI (note the hyphen), at commit 20535d8f.

One detail made detection harder: commit 20535d8f existed only on the attacker's fork, yet it opened fine through the parent repository's URL, because GitHub serves a fork's commits through the repository it was forked from.

Staging finished at 11:08:09, and the production pipeline started at 11:08:53.

Step 5 — The production gate approved with a stolen token (11:11–11:46)

At 11:11:18, the production environment gate was approved with the second maintainer's token, from IP 199.91.221.39. The pipeline signed and published v1.107.0, built from the commit on the fork.

• 11:39:51 – 11:40:13: seven wheels to PyPI
• 11:46:56: Docker Hub and GHCR images

Steps 6–7 — Covering the tracks (11:53–12:24)

• 11:53:49 and 11:54:01: the third maintainer's token deleted the production then staging run records, taking all workflow logs with them
• 12:09:38: revert commit 285fbdc5 was pushed with the owner token, overriding branch protection again
• 12:24:52: main was force-pushed back to d687b69, the commit before the change, bypassing the force-push block as well After that, looking at main showed neither the change nor the revert. Both commits remain retrievable by SHA, however, and the organization's audit log recorded each override.

Why the approval process did not save them

This is the part every DevOps team should read carefully.

In normal operation, building and publishing a release required:

• Being initiated by an authorized engineer
• Being approved by a dedicated account tied to a physical device kept in the office That is a strong process. But according to the RCA, the restriction had just been relaxed that same week to support a temporary need for more frequent releases and patch backports, allowing any two engineers to trigger and approve a release.

The attacker used exactly two different tokens for those two roles.

LiteLLM adds a sharp observation: with the org owner's token in hand, the attacker could have relaxed or removed that restriction themselves. They did not. LiteLLM assesses that the reason was to avoid generating noisy audit log entries.

The payload in 1.107.0

According to Microsoft Security Research and other reporting, the files bundled with 1.107.0 were designed to execute automatically when Python starts, even if the application never imports LiteLLM.

Two execution branches:

1. A credential harvesting module, targeting:

• Development tools, AI applications, cloud environments and browsers
• System information, SSH files, Git and GitLab credentials and tokens
• Environment variables
• Tokens for npm, PyPI, pip, Yarn, Maven, Gradle, Composer, RubyGems and Cargo The data was compressed and POSTed to litellm[.]dev/healthcheck. That domain was registered on October 13, eight days before the release shipped.

2. A Unix-oriented backdoor, running as a background process and pulling and executing arbitrary attacker code every hour.

The most important warning for remediation, from Microsoft's analysis: because the startup files live outside LiteLLM's application directory, deleting only the application files may leave the malicious startup components behind.

Assessment

There is no software vulnerability anywhere in this chain. Every link was an ordinary operational decision: committing a config file by mistake, forgetting to revoke, configuring a token the convenient way, relaxing a process to move faster. That is what makes this incident more worth reading than most CVE reports: there is nothing to patch.

GitHub CLI tokens are a dangerously underrated class of credential. Many teams treat "it's behind 2FA" as sufficient. But that is precisely what makes it dangerous: unscoped, already past 2FA, and not stopped by the re-auth flow for sensitive operations. A repo-scoped PAT in the same position would have made step 1 a dead end.

The "temporary" relaxation of the approval process landed in the exact week of the attack. It is worth being direct about this: requiring approval from a physical device in the office is a strong control, and it was loosened for a perfectly reasonable purpose, shipping patches faster. This is the kind of risk no security checklist catches, because the decision to relax always looks reasonable at the moment it is made.

The attacker understood audit logs well. They split the work across three identities, used branches that lived minutes, deleted run records, and overrode branch protection only when strictly necessary. Choosing not to remove the approval restriction despite having the authority to do so shows the priority was reducing signal, not reducing steps.

The most worrying entry in the stolen secret list is GITGUARDIAN_API_KEY. As Socket points out, this is a credential for the secret-scanning platform itself, giving the attacker visibility into, and potentially influence over, the very alerts that should have caught this activity. In the same category is E2E_LINEAR_STORAGE_STATE_B64, a Playwright storage state that grants Linear account access without a password and without clearing MFA.

Detection came from the community, not from monitoring. Lidor Machluf (Upwind Security) opened the issue identifying the compromised release at 12:06:41, roughly 20 minutes after the package hit PyPI. LiteLLM began its response at 13:02. The malicious artifacts were live on the registries for about two hours.

Relevance for Vietnam

• LiteLLM is widely used by internal AI teams in Vietnam, usually as a multi-model gateway. This is not an incident to read for reference, it is one to check directly: did any host in your organization install 1.107.0 on October 21.
• Committing config files to repos is a widespread habit, especially in fast-moving internal projects. The gap between "detected and noted for revocation" and "actually revoked" is where this incident was born.
• Unscoped personal GitHub tokens are still common in CI at smaller teams, because they are convenient and because scoped PATs tend to fail on permissions mid-run.
• Developer machines are credential concentrators. This payload harvested tokens for nearly every common package manager, so one infected dev host means a broad credential rotation, not a single uninstall.
• Your own release pipeline deserves a review, specifically three questions: can the release workflow accept a source repository, who can override branch protection, and is any approval requirement currently in a "temporarily relaxed" state.

Recommendations

If you ran an affected version

• Treat every credential on that host or container as exposed and rotate it: SSH keys, Git and GitLab tokens, every package manager token, model provider API keys, environment variables, cloud credentials.
• Do not stop at removing the package. The startup components live outside the application directory, so the Python auto-start mechanisms in that environment must be inspected too. For containers, rebuilding from a verified image is the clean path.
• Hunt for the backdoor: outbound connections on an hourly cadence, and any request to litellm[.]dev.
• Review logs from around October 21 onward on every system that host could reach.

Process lessons for DevOps and security teams

• Turn key revocation into a ticket with an owner and a deadline. "Noted for revocation" is not a terminal state. If your organization has a list of once-exposed keys that were never rotated, that list is this week's work.
• Deleting a file does not delete Git history. Once a secret reaches a public repo, assume it has been collected, no matter how fast you removed it.
• Never put an unscoped token into an automated service. Every automated system needs the narrowest credential possible, limited by repository and by action, and ideally short-lived.
• Lock down the build source of your release pipeline. If a release workflow accepts a source repository as an input, that is a design flaw. If it hardcodes one, protect the workflow file itself with codeowners and let nobody override it.
• Review who can override branch protection and force-push. In this incident, the org owner's override authority is what reduced every remaining control to a formality. Consider enabling admin enforcement on critical branches.
• Treat relaxing an approval process as a risk-bearing change, one that is recorded, time-boxed, and alerts when the window expires without restoration. "Temporary" relaxations that are never restored recur across a great many incidents.
• Monitor CI for anomalous behavior: very short-lived branches following a fixed naming pattern, workflows that appear and vanish, deleted run records, and secrets read by a workflow unrelated to its function.

IOCs

From the Indicators section of LiteLLM's RCA, supplemented by Socket's analysis via The Hacker News.

IP addresses

69.48.229.143      # Agent Platform API calls on Oct 10
199.91.221.39      # pushes to BerriAI/project-releaser and production gate approval on Oct 21

Exfiltration infrastructure

webhook[.]site/65a42464-3cfa-4b02-a18c-17a6d7b86f37
webhook[.]site/4da03473-a3e7-4dc2-a087-9c564b4208e1
litellm[.]dev/healthcheck          # payload destination, domain registered 2026-10-13

GitHub organization and commits

Berri-AI                # lookalike organization (note the hyphen), hosting the fork used as build source
20535d8f                # commit on the fork the release was built from
9cf97e9                 # commit adding the source_owner input to the release workflow
285fbdc5                # revert commit, later force-pushed off main
d687b69                 # commit main was force-pushed back to

Throwaway exfiltration branches

ci-tmp-a351cc           # BerriAI/litellm
ci-tmp-be33e9           # BerriAI/litellm-auto
ci-tmp-4d8517           # second wave, per Socket

Repository indicators

File:     .github/workflows/cache.yml      (does not belong to the repo)
Trigger:  on: [push]
Subject:  "ci: bump cache config"
Pattern:  ci-tmp-<hex> branch cut from a legitimate commit, workflow runs, branch deleted

Affected artifacts

PyPI:          litellm==1.107.0   (7 wheels, published 11:39:51-11:40:13 UTC on Oct 21)
Docker Hub:    v1.107.0, latest, main-latest, main-stable
GHCR:          equivalent tags
Exposure:      roughly 2 hours before removal

References

• LiteLLM (Oliver Jensen, Mateo Wang) — The attack chain behind the v1.107.0 LiteLLM release (Oct 23, 2026): https://docs.litellm.ai/blog/security-incident-oct-21-2026-attack-chain
• The Hacker News — Leaked LiteLLM Master Key Let Attackers Steal GitHub Tokens and Hijack Release Pipeline: https://thehackernews.com/2026/10/leaked-litellm-master-key-let-attackers.html
• LiteLLM-Labs/lite-harness — the mistakenly committed configuration file: https://github.com/LiteLLM-Labs/lite-harness/blob/e10b14a/harnesses/opencode/opencode.json
• Socket (Karlo Zanki) — analysis of the malicious workflows used for exfiltration
• Microsoft Security Research — behavioral analysis of the litellm 1.107.0 package
• Lidor Machluf (Upwind Security) — issue #47826 identifying the compromised release