LiteLLM has traced last week’s supply chain attack to a master key it accidentally committed to a public GitHub repository in May. The cloud and AI model secret-stealing part of the incident began when someone uploaded a poisoned version (1.107.0) of the popular open source AI gateway to PyPI and its container registries last Wednesday. This compromised version contained an infostealer and a remote access trojan capable of executing arbitrary code on infected machines.

While the trojanized package was available on PyPI for less than two hours before being quarantined, it received about 49,000 downloads during that time, according to Wiz. On Friday, LiteLLM detailed the attack chain behind the compromised release and traced it back to May 27, when a configuration file containing a master key was committed to the public LiteLLM-Labs/lite-harness repository. The key belonged to an open source tool used internally by LiteLLM to orchestrate AI agents.

The project's developers quickly spotted the error and removed the exposed commit, but failed to revoke the master key. “On October 10th, the threat actor used this key to gain access to our LiteLLM Agent Platform and extracted this GitHub CLI OAuth token,” according to Friday’s postmortem. “From this point onwards, the threat actor had owner-level access to all repositories in our GitHub.”This is especially serious because GitHub CLI tokens are “post-2fa,” meaning that they don’t trigger GitHub's sudo mode authentication flows typically required to perform high-impact repository actions.

The token allowed the attacker to author malicious commits into branches on other internal repos - including workflows designed to steal any tokens and secrets configured to be available during the continuous integration development process. And this allowed them to steal other GitHub CLI OAuth tokens linked to two engineers with maintainer privileges. Plus, they acquired tokens for several other internal systems, we’re told.

“Reviewing the actor's activity on these other systems shows attempts to identify and exfiltrate additional keys and tokens, with a primary focus on upstream inference API tokens,” LiteLLM noted. Having now stolen several privileged identities, the miscreant could make changes to GitHub repositories without raising any suspicion because the actions and approvals all appeared to come from legitimate, credentialed engineers. In response, LiteLLM’s developers are revoking every token associated with every internal system and vendor platform.

“This incident has provided a strong illustration of the risks associated with failing to rotate a potentially exposed key,” they wrote. The infostealers hidden inside the compromised package were designed to hoover up a ton of sensitive information, including browser credentials and cookies from Chrome, Firefox, Edge, and Safari, along with AWS credentials, Google Cloud Platform service account keys and application default credentials, and Microsoft Azure tokens. It also swiped API keys for OpenAI, Anthropic, Cohere, and other LLM providers.

So if you installed and ran version 1.107.0, assume any credentials accessible to the affected system may have been stolen. Investigate affected systems, change passwords, and rotate exposed keys. A quick prayer to the patron saint of supply chains wouldn't hurt, either. ®