# One key nobody revoked, four months later a malicious PyPI package ## Summary On **October 21, 2026**, LiteLLM's release pipeline published a malicious `litellm==1.107.0` to **PyPI, Docker Hub and GHCR**. The package was signed and shipped through the official process, so every signature check passed. The root cause was not a software vulnerability. It was **an administrative key committed to a public repo on May 27, 2026 and never revoked**. More than four months later, an attacker used that key to obtain an org owner's GitHub CLI token, then kept going until they could sign a release. The most dangerous property of the payload: the bundled files **execute automatically when Python starts**, even if the application never does `import litellm`. **Priority actions:** - **If you installed** `litellm==1.107.0` **or pulled** `v1.107.0`**,** `latest`**,** `main-latest` **or** `main-stable` **on October 21:** treat every credential on that host as exposed, and note that the **startup files live outside the application directory**, so removing the package alone does not clean it. - **Audit every key that was once committed by mistake and then "noted for revocation".** This incident exists because exactly one follow-up task went undone. ## What the leaked file looked like The file was `harnesses/opencode/opencode.json` in the public repo `LiteLLM-Labs/lite-harness`, a configuration file for an agent harness. Four things stand out from its structure alone, without knowing any key value: 1. **The same** `sk-…` **key was hardcoded in four separate places** in one file: in the `Authorization: Bearer` header of three remote MCP servers pointing at an internal gateway, and in the provider's `apiKey` field. One key serving several purposes means revoking it breaks several things at once, and that is exactly the kind of friction that makes people postpone revocation. 2. **An API key for a third-party sandbox service** sat in an environment variable. 3. **Placeholders such as** `sk-dev-master-key-change-me` for a LAP endpoint on `localhost`. In other words, this was a development config with sample values and real values mixed together. 4. **An absolute path on a developer's machine** was exposed in the `command` section. Nothing about the file is sophisticated. It is what happens every day in every organization: a dev config committed for convenience, with one real value hidden among the samples. ## Four silent months According to LiteLLM's RCA, the mistake was **caught quickly**. Roughly three hours later a follow-up commit removed the file, and the exposed keys were **noted for revocation**. The revocation never happened. LiteLLM states it plainly as human error: *"nobody followed through on revoking the administrative key"*. One more detail matters: **the original commit remained in the repo's public history until October 22**, a day after the incident broke. Deleting a file in a later commit does not remove its contents from Git history. > **Source note:** The Hacker News reports that the commit was "force-pushed over". LiteLLM's official RCA says otherwise: a follow-up commit removed the file, and the original commit stayed in public history. This post follows the RCA. The key remained valid for more than four months. ## The seven-step chain All timestamps below are UTC, taken from the detailed timeline in the RCA. ### Step 1 — An old key traded for a GitHub token (Oct 10, 21:13–21:28) From IP `69.48.229.143`, the attacker called the LiteLLM Agent Platform API using the mistakenly committed administrative key. The platform was an internal experimental project that let self-hosted cloud agents open pull requests on GitHub. At **21:27:10**, a `GET /api/v1/managed_agents/agents?limit=100` request returned the configuration of **every managed agent**. One of them held **a GitHub CLI OAuth token belonging to one of the two org owners**. ### The pivotal detail: what kind of token was stolen This is the part worth dwelling on longest. The token configured in the Agent Platform was **not a Personal Access Token scoped to specific repositories and capabilities**. It was a **GitHub CLI OAuth token tied to an org owner account**. As LiteLLM describes it, this type of token is: - **Unscoped** - **Behind 2FA**, meaning two-factor authentication has already been cleared - **Not subject to GitHub's "super-user" re-auth flow** for sensitive organization- or repository-level operations From this point on, the attacker had **owner-level access to the entire GitHub organization**. ### Steps 2–3 — The victim's own CI becomes the exfiltration channel (Oct 10 and Oct 12) The attacker needed no new tooling. They used GitHub Actions. The pattern, described in detail by Socket: 1. Create a throwaway branch named `ci-tmp-` **from a recent legitimate commit** so the push looks unremarkable 2. Push a commit adding `.github/workflows/cache.yml`, with an equally unremarkable subject: *"ci: bump cache config"* 3. The workflow is configured `on: [push]`, so it **runs immediately** 4. Delete the branch Results: | Time | Branch | Repo | Obtained | | --- | --- | --- | --- | | 22:11:45 | `ci-tmp-a351cc` | `BerriAI/litellm` | 6 secrets, including `GH_TOKEN`, the GitHub CLI token of a **second maintainer** | | 22:12:09 | `ci-tmp-be33e9` | `BerriAI/litellm-auto` | `RELEASE_PLEASE_TOKEN`, the GitHub CLI token of a **third maintainer** | Both run records were deleted within three minutes, at 22:14:14 and 22:14:31. On **October 12**, a second wave expanded into several other private repositories. According to Socket's analysis, the harvest covered GitHub tokens and API keys for Linear, a LiteLLM deployment, Together AI, plus a GitHub App's identity and signing key; the later wave added keys for Anthropic, OpenAI, AWS Bedrock, Tavily, **a GitGuardian secret-scanning tenant**, along with a Playwright storage state, PostgreSQL connection details and a password-bearing Redis endpoint. The data went to two different `webhook[.]site` endpoints. ### Step 4 — Repointing the release pipeline at a fork (Oct 21, 10:08–11:08) The release workflows live in the private repo `BerriAI/project-releaser`, and they **hardcoded** `BerriAI` **as the source owner**, so no input allowed building from a fork. To build from one, the attacker had to modify the workflow itself. They authored commit `9cf97e9` (*"ci: release from a fork source repository"*), adding a `source_owner` input that defaults to `BerriAI` but accepts any organization. Then came the five most instructive seconds of the whole incident: ```plaintext 10:08:29 Push the commit with the second maintainer's token -> BLOCKED by branch protection (pull request required, 2 passing checks, codeowner approval) 10:08:34 Push again with the org owner's token -> SUCCEEDED, branch protection overridden ``` At **10:46:39**, the staging workflow was dispatched with its source pointing at `Berri-AI/litellm`, a fork inside the lookalike organization `Berri-AI` (note the hyphen), at commit `20535d8f`. One detail made detection harder: commit `20535d8f` existed only on the attacker's fork, **yet it opened fine through the parent repository's URL**, because GitHub serves a fork's commits through the repository it was forked from. Staging finished at 11:08:09, and the production pipeline started at 11:08:53. ### Step 5 — The production gate approved with a stolen token (11:11–11:46) At **11:11:18**, the production environment gate was approved with the second maintainer's token, from IP `199.91.221.39`. The pipeline signed and published v1.107.0, built from the commit on the fork. - **11:39:51 – 11:40:13:** seven wheels to PyPI - **11:46:56:** Docker Hub and GHCR images ### Steps 6–7 — Covering the tracks (11:53–12:24) - **11:53:49** and **11:54:01:** the third maintainer's token deleted the production then staging run records, taking all workflow logs with them - **12:09:38:** revert commit `285fbdc5` was pushed with the owner token, overriding branch protection again - **12:24:52:** `main` was **force-pushed back to** `d687b69`, the commit before the change, bypassing the force-push block as well After that, looking at `main` showed neither the change nor the revert. Both commits remain retrievable by SHA, however, and **the organization's audit log recorded each override**. ## Why the approval process did not save them This is the part every DevOps team should read carefully. In normal operation, building and publishing a release required: - Being initiated by an authorized engineer - Being approved by **a dedicated account tied to a physical device kept in the office** That is a strong process. But according to the RCA, the restriction **had just been relaxed that same week** to support a temporary need for more frequent releases and patch backports, allowing **any two engineers** to trigger and approve a release. The attacker used exactly two different tokens for those two roles. LiteLLM adds a sharp observation: with the org owner's token in hand, the attacker **could have relaxed or removed that restriction themselves**. They did not. LiteLLM assesses that the reason was **to avoid generating noisy audit log entries**. ## The payload in 1.107.0 According to Microsoft Security Research and other reporting, the files bundled with 1.107.0 were designed to **execute automatically when Python starts**, even if the application never imports LiteLLM. Two execution branches: **1\. A credential harvesting module**, targeting: - Development tools, AI applications, cloud environments and browsers - System information, SSH files, Git and GitLab credentials and tokens - Environment variables - Tokens for npm, PyPI, pip, Yarn, Maven, Gradle, Composer, RubyGems and Cargo The data was compressed and POSTed to `litellm[.]dev/healthcheck`. **That domain was registered on October 13**, eight days before the release shipped. **2\. A Unix-oriented backdoor**, running as a background process and **pulling and executing arbitrary attacker code every hour**. The most important warning for remediation, from Microsoft's analysis: because **the startup files live outside LiteLLM's application directory**, deleting only the application files may **leave the malicious startup components behind**. ## Assessment **There is no software vulnerability anywhere in this chain.** Every link was an ordinary operational decision: committing a config file by mistake, forgetting to revoke, configuring a token the convenient way, relaxing a process to move faster. That is what makes this incident more worth reading than most CVE reports: there is nothing to patch. **GitHub CLI tokens are a dangerously underrated class of credential.** Many teams treat "it's behind 2FA" as sufficient. But that is precisely what makes it dangerous: unscoped, already past 2FA, and not stopped by the re-auth flow for sensitive operations. A repo-scoped PAT in the same position would have made step 1 a dead end. **The "temporary" relaxation of the approval process landed in the exact week of the attack.** It is worth being direct about this: requiring approval from a physical device in the office is a strong control, and it was loosened for a perfectly reasonable purpose, shipping patches faster. This is the kind of risk no security checklist catches, because the decision to relax always looks reasonable at the moment it is made. **The attacker understood audit logs well.** They split the work across three identities, used branches that lived minutes, deleted run records, and overrode branch protection only when strictly necessary. Choosing not to remove the approval restriction despite having the authority to do so shows the priority was **reducing signal**, not reducing steps. **The most worrying entry in the stolen secret list is** `GITGUARDIAN_API_KEY`**.** As Socket points out, this is a credential for the secret-scanning platform itself, giving the attacker visibility into, and potentially influence over, the very alerts that should have caught this activity. In the same category is `E2E_LINEAR_STORAGE_STATE_B64`, a Playwright storage state that grants Linear account access **without a password and without clearing MFA**. **Detection came from the community, not from monitoring.** Lidor Machluf (Upwind Security) opened the issue identifying the compromised release at 12:06:41, roughly **20 minutes after the package hit PyPI**. LiteLLM began its response at 13:02. The malicious artifacts were live on the registries for about two hours. ## Relevance for Vietnam - **LiteLLM is widely used by internal AI teams in Vietnam**, usually as a multi-model gateway. This is not an incident to read for reference, it is one **to check directly**: did any host in your organization install 1.107.0 on October 21. - **Committing config files to repos is a widespread habit**, especially in fast-moving internal projects. The gap between "detected and noted for revocation" and "actually revoked" is where this incident was born. - **Unscoped personal GitHub tokens are still common in CI** at smaller teams, because they are convenient and because scoped PATs tend to fail on permissions mid-run. - **Developer machines are credential concentrators.** This payload harvested tokens for nearly every common package manager, so one infected dev host means a broad credential rotation, not a single uninstall. - **Your own release pipeline deserves a review**, specifically three questions: can the release workflow accept a source repository, who can override branch protection, and is any approval requirement currently in a "temporarily relaxed" state. ## Recommendations ### If you ran an affected version - **Treat every credential on that host or container as exposed** and rotate it: SSH keys, Git and GitLab tokens, every package manager token, model provider API keys, environment variables, cloud credentials. - **Do not stop at removing the package.** The startup components live **outside the application directory**, so the Python auto-start mechanisms in that environment must be inspected too. For containers, rebuilding from a verified image is the clean path. - **Hunt for the backdoor:** outbound connections on an **hourly** cadence, and any request to `litellm[.]dev`. - **Review logs from around October 21** onward on every system that host could reach. ### Process lessons for DevOps and security teams - **Turn key revocation into a ticket with an owner and a deadline.** "Noted for revocation" is not a terminal state. If your organization has a list of once-exposed keys that were never rotated, that list is this week's work. - **Deleting a file does not delete Git history.** Once a secret reaches a public repo, assume it has been collected, no matter how fast you removed it. - **Never put an unscoped token into an automated service.** Every automated system needs the narrowest credential possible, limited by repository and by action, and ideally short-lived. - **Lock down the build source of your release pipeline.** If a release workflow accepts a source repository as an input, that is a design flaw. If it hardcodes one, protect the workflow file itself with codeowners and let nobody override it. - **Review who can override branch protection and force-push.** In this incident, the org owner's override authority is what reduced every remaining control to a formality. Consider enabling admin enforcement on critical branches. - **Treat relaxing an approval process as a risk-bearing change**, one that is recorded, time-boxed, and alerts when the window expires without restoration. "Temporary" relaxations that are never restored recur across a great many incidents. - **Monitor CI for anomalous behavior:** very short-lived branches following a fixed naming pattern, workflows that appear and vanish, deleted run records, and secrets read by a workflow unrelated to its function. ## IOCs > From the Indicators section of LiteLLM's RCA, supplemented by Socket's analysis via The Hacker News. **IP addresses** ```plaintext 69.48.229.143 # Agent Platform API calls on Oct 10 199.91.221.39 # pushes to BerriAI/project-releaser and production gate approval on Oct 21 ``` **Exfiltration infrastructure** ```plaintext webhook[.]site/65a42464-3cfa-4b02-a18c-17a6d7b86f37 webhook[.]site/4da03473-a3e7-4dc2-a087-9c564b4208e1 litellm[.]dev/healthcheck # payload destination, domain registered 2026-10-13 ``` **GitHub organization and commits** ```plaintext Berri-AI # lookalike organization (note the hyphen), hosting the fork used as build source 20535d8f # commit on the fork the release was built from 9cf97e9 # commit adding the source_owner input to the release workflow 285fbdc5 # revert commit, later force-pushed off main d687b69 # commit main was force-pushed back to ``` **Throwaway exfiltration branches** ```plaintext ci-tmp-a351cc # BerriAI/litellm ci-tmp-be33e9 # BerriAI/litellm-auto ci-tmp-4d8517 # second wave, per Socket ``` **Repository indicators** ```plaintext File: .github/workflows/cache.yml (does not belong to the repo) Trigger: on: [push] Subject: "ci: bump cache config" Pattern: ci-tmp- branch cut from a legitimate commit, workflow runs, branch deleted ``` **Affected artifacts** ```plaintext PyPI: litellm==1.107.0 (7 wheels, published 11:39:51-11:40:13 UTC on Oct 21) Docker Hub: v1.107.0, latest, main-latest, main-stable GHCR: equivalent tags Exposure: roughly 2 hours before removal ``` ## References - LiteLLM (Oliver Jensen, Mateo Wang) — *The attack chain behind the v1.107.0 LiteLLM release* (Oct 23, 2026): - The Hacker News — *Leaked LiteLLM Master Key Let Attackers Steal GitHub Tokens and Hijack Release Pipeline*: - LiteLLM-Labs/lite-harness — the mistakenly committed configuration file: - Socket (Karlo Zanki) — analysis of the malicious workflows used for exfiltration - Microsoft Security Research — behavioral analysis of the litellm 1.107.0 package - Lidor Machluf (Upwind Security) — issue #47826 identifying the compromised release