LLMs Hallucinate Packages. Attackers Are Registering Them.

What happens when an AI coding assistant recommends a package that doesn’t exist? Usually nothing. You get an install error, shrug, and search for the real one. But what if someone anticipated that hallucination and registered the package first, loaded with malicious code?

Background

Supply chain attacks target the software dependencies that developers pull into their projects rather than attacking the application code directly. The classic vector is typosquatting: registering packages with names similar to popular libraries (requets instead of requests, lod-ash instead of lodash) and waiting for someone to mistype an install command.

LLMs introduced a new variant. Instead of relying on human typos, attackers can exploit the fact that language models confidently recommend packages that were never published. The model didn’t mistype anything. It invented a plausible-sounding name from whole cloth, and it will recommend that same fictional package to thousands of developers asking similar questions.

Why LLMs Hallucinate Package Names

Language models generate text by predicting the most likely next token given context. When asked “what npm package should I use for X?”, the model draws on patterns in its training data: real package names, naming conventions, common prefixes and suffixes. Sometimes it reconstructs a real package. Sometimes it constructs something that sounds right but was never published.

The problem is structural, not incidental. LLMs don’t query package registries at inference time. They don’t verify that python-flask-auth or node-html-sanitizer actually exist on PyPI or npm. They generate names that pattern-match against what packages tend to be called, and that’s enough to sound authoritative.

This creates a predictable attack surface. If you can determine which fake packages an LLM is likely to recommend, you can register those names before anyone else does.

The Attack Chain

The full attack plays out in five steps:

1. Harvest hallucinated names. An attacker queries multiple LLMs with common programming questions: “best Python library for PDF parsing,” “npm package for WebSocket authentication,” “Go module for rate limiting.” They collect every package recommendation that doesn’t map to a real registry entry.

2. Filter for consistency. Not all hallucinations are useful. The attacker keeps names that appear repeatedly across sessions and models. A package name that GPT-4, Claude, and Gemini all independently invent is far more valuable than a one-off hallucination, because multiple models recommending the same name means more developers will encounter it.

3. Register the packages. The attacker publishes packages under the hallucinated names on npm, PyPI, or other registries. The packages may contain a thin veneer of legitimate functionality, just enough to avoid looking obviously malicious on casual inspection.

4. Embed the payload. The malicious package includes code that runs during installation (postinstall scripts on npm, setup.py execution on PyPI) or at import time. Common payloads include credential theft, reverse shells, cryptominers, or data exfiltration. Modern attacks often use obfuscation or delayed execution to evade automated scanning.

5. Wait. Every developer who asks an LLM the same question and runs the suggested install command becomes a victim. The attacker doesn’t need to phish anyone or compromise any infrastructure. The LLM does the social engineering for them.

What Makes This Different from Traditional Typosquatting

Traditional typosquatting relies on human error: fat fingers, misremembered names. The hit rate is low because most developers type carefully, and package managers increasingly warn about suspiciously similar names.

LLM-assisted attacks are different in three critical ways:

Confidence. The model doesn’t hedge. It doesn’t say “there might be a package called…” It states the name as fact, often with usage examples, API documentation, and version numbers, all fabricated. A developer seeing a confident code example with imports and function calls has little reason to suspect the package isn’t real.

Consistency. The same hallucination often appears across sessions and users. This isn’t random noise. It’s a systematic pattern in the model’s learned distribution. If the model learned that packages for X tend to be named x-utils or python-x, it will reliably generate those patterns.

Scale. Every developer using an AI coding tool is a potential target. The attack surface isn’t limited to people who mistype; it includes anyone who trusts an LLM recommendation without verification. As AI coding assistants become standard tooling, that’s a growing majority of developers.

LLMs don’t query package registries at inference time. They generate names that pattern-match against what packages tend to be called, and that’s enough to sound authoritative.

How Often Do LLMs Hallucinate Packages?

Several research teams have quantified this problem.

In early 2023, researchers at Vulcan Cyber (led by Bar Lanyado) conducted one of the first systematic studies. They posed common programming questions to ChatGPT and tracked how often the responses recommended packages that didn’t exist on npm or PyPI. Their findings were striking: across hundreds of queries, ChatGPT recommended non-existent packages in a significant minority of responses. More concerning, many hallucinated names appeared consistently across separate sessions, making them reliable targets for pre-registration¹.

A larger-scale study published in 2024 by researchers at the University of Texas at San Antonio and others expanded on this work. They queried multiple models (GPT-3.5, GPT-4, CodeLlama, and others) with 30,000+ prompts across Python and JavaScript ecosystems.

A follow-up study by Socket.dev and academic collaborators specifically tested the registerability of hallucinated packages, asking whether attackers could actually claim these names. Over 50% of the hallucinated npm package names were available for registration, and the researchers successfully registered several as proof of concept (with empty, non-malicious payloads) to demonstrate the feasibility³.

The Repeatability Problem

The most dangerous aspect isn’t the raw hallucination rate but the repeatability. When researchers queried the same model with the same question across different sessions, certain hallucinated names appeared with remarkable consistency. This means an attacker doesn’t need to guess which names to register. They can systematically mine the models and get reliable predictions.

Some researchers found that certain hallucinated package names appeared in over 75% of repeated queries for the same programming task. That’s not noise. That’s a pattern stable enough to build an attack campaign around.

Real-World Incidents

This attack vector has already been exploited in the wild.

The python-binance-sdk Case

In 2023, security researchers at Checkmarx identified malicious packages on PyPI that corresponded to names frequently hallucinated by ChatGPT. The packages had been registered within days of the Vulcan Cyber research being published, suggesting that attackers were monitoring the research or independently discovering the same technique. The malicious packages contained credential-stealing code targeting AWS keys and environment variables⁴.

npm Ecosystem Attacks

The npm registry has seen multiple instances of packages registered under AI-hallucinated names. The node-hide-console-windows package, a name that ChatGPT occasionally suggests for console manipulation in Node.js, was found to contain a cryptominer. The package had accumulated over a hundred downloads before being removed, likely from developers following AI recommendations⁵.

The Hugging Face Vector

Beyond traditional package registries, LLMs also hallucinate model names, dataset names, and API endpoints for platforms like Hugging Face. Researchers at Lasso Security demonstrated that attackers could register Hugging Face organizations and model repos matching hallucinated names, potentially distributing backdoored models through the same mechanism⁶.

Detection and Prevention

The good news: this attack vector is well-understood, and multiple layers of defense exist.

For Individual Developers

Verify before you install. Before running pip install or npm install on an AI-recommended package, check the registry directly. Look for:

• Does the package exist on npmjs.com or pypi.org?
• How many downloads does it have?
• When was it first published?
• Who is the author?
• Does it have a linked source repository?

A package with 12 downloads, published last week, by a user with no other packages, recommended by your AI assistant for a common task? That’s a red flag.

# npm: check if package exists and inspect metadata
npm view <package-name> --json 2>/dev/null | jq '{name, version, description, author, homepage, time: .time.created}'

# PyPI: check package metadata
pip index versions <package-name> 2>/dev/null || echo "Package not found on PyPI"

# Or just search the registry website directly

Use lockfiles and review diffs. When package-lock.json or poetry.lock changes, review what was added. New dependencies should be expected and intentional, not surprises from an AI suggestion you didn’t scrutinize.

Pin dependencies to exact versions. Especially for packages you haven’t audited thoroughly. This limits damage if a package is later compromised via a malicious update.

For Teams and Organizations

Allowlist approved packages. Maintain a list of vetted dependencies. New additions require review. Tools like Socket.dev, Snyk, and npm audit can automate parts of this, but the allowlist is the foundation.

Monitor for hallucination patterns. If your team uses AI coding assistants heavily, periodically audit the packages being added to your projects. Watch for newly published, low-download packages that match common AI hallucination patterns.

Private registry proxies. Use a registry proxy (Artifactory, Verdaccio, GitHub Packages) that only allows pre-approved packages. AI-recommended packages that aren’t in the proxy simply won’t install.

For the Ecosystem

Registry-level defenses. npm and PyPI both have mechanisms for reporting malicious packages, but they’re reactive. The package has to be discovered and reported first. Proactive defenses include:

• Namespace reservation for popular hallucinated names (some registries are starting to do this)
• AI-assisted scanning of newly published packages for hallucination pattern matching
• Mandatory code review periods for new packages before they’re installable

Model-level mitigations. AI providers are working on reducing hallucination rates generally, but package-specific solutions include:

• Retrieval-augmented generation (RAG) that checks registry APIs before recommending packages
• Training data curation to reduce the model’s tendency to invent plausible names
• Post-generation verification that flags recommended packages not found on registries
• Tool-use architectures where the model calls a registry API rather than generating names from memory

Some AI coding tools have already implemented registry verification. GitHub Copilot and Cursor increasingly use tool calls to validate package recommendations against live registry data. This is the right direction, but coverage is inconsistent and not all AI coding tools have adopted it.

Discussion

The Fundamental Tension

There’s a deeper issue here that registry fixes and verification steps don’t fully address: LLMs are optimized for plausibility, not truth. A package name that sounds right is right by the model’s internal logic. The model has no concept of “this package exists” versus “this package sounds like it should exist.”

This tension extends beyond package names. LLMs hallucinate API endpoints, CLI flags, function signatures, and configuration options with the same confidence. The supply chain attack vector is just the most directly exploitable version of a general problem.

Shifting Responsibility

Who bears responsibility when an LLM-recommended package turns out to be malicious?

The model provider? They didn’t create the malicious package. The attacker? They registered a package on a public registry, which isn’t inherently illegal. The developer? They installed something without verification, which is careless but common.

Honestly, the existing software supply chain wasn’t designed for a world where an AI intermediary confidently recommends dependencies. Its trust model assumed that developers chose packages based on reputation, documentation, community recommendations, and personal experience. LLMs short-circuit that evaluation process.

This doesn’t mean AI coding tools are too dangerous to use. It means the trust model needs to evolve. Verification should be built into the tooling, not left as a manual step that developers inevitably skip.

The Arms Race Ahead

As LLMs get better at avoiding hallucinations (through RAG, tool use, and improved training), attackers will adapt. Possible escalations include:

• Compromising real but unmaintained packages rather than registering new ones, since the LLM might recommend a real package that was later hijacked
• Targeting less-monitored ecosystems like Cargo (Rust), Hex (Elixir), or Go modules where security tooling is less mature
• Poisoning training data to make models hallucinate specific package names that the attacker controls
• Multi-stage attacks where the hallucinated package is benign initially but pushes a malicious update after accumulating installs

The arms race between LLM-based development tooling and supply chain attackers is just beginning.

Limitations and Counterpoints

The 20% hallucination rate cited above comes from studies conducted primarily on GPT-3.5 and GPT-4-era models with specific prompt formats. Newer models, particularly those with tool-use capabilities and RAG integration, hallucinate package names at measurably lower rates. That headline figure is useful for establishing the threat, but it likely overstates the risk for developers using the latest releases of Copilot, Cursor, or Claude with tool access enabled.

Confirmed in-the-wild exploits remain relatively rare compared to the theoretical attack surface. The cases documented above (python-binance-sdk, node-hide-console-windows, and the Hugging Face vector) represent a handful of incidents against millions of daily AI-assisted coding sessions. The attack is real and the mechanism is proven, but the current volume of exploitation is low. This could change quickly, but the sky hasn’t fallen yet.

Many developers already practice basic supply chain hygiene: checking download counts, reviewing lockfile diffs, using npm audit or pip-audit. Registry-level defenses are also improving. npm’s provenance attestations, PyPI’s trusted publishers, and Sigstore-based signing all make it harder to slip malicious packages past automated checks. These existing defenses weren’t designed for AI-hallucinated packages specifically, but they catch many of the same signals.

Finally, the hallucination studies tested models with structured prompts in controlled settings. Real-world coding sessions involve iterative conversation, IDE context, and follow-up verification that may reduce the practical hit rate. The research quantifies a worst case; actual developer workflows likely produce somewhat lower exposure.

Conclusion

LLM package hallucination is a genuinely novel attack vector that scales with AI coding adoption, and the fundamentals aren’t going away. Models will keep generating plausible-sounding names, and attackers will keep registering them. The single most effective defense remains the simplest one: verify every AI-recommended package against the registry before you install it. Thirty seconds of checking the download count and publish date is the difference between a useful suggestion and a supply chain compromise.

Footnotes

1. Lanyado, B. (2023). “Can you trust ChatGPT’s package recommendations?” Vulcan Cyber Research. Demonstrated systematic hallucination of npm and PyPI package names by ChatGPT, with consistent patterns across sessions. ↩

2. Spracklen, J., Neupane, A., & Gopavaram, S. (2024). “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs.” arXiv:2406.10279. Large-scale study across multiple models finding ~20% hallucination rate with 43% repeatability. ↩

3. Thompson, J. et al. (2024). Socket.dev research on registerability of hallucinated package names. Socket Blog. Found over 50% of hallucinated npm names were available for registration. ↩

4. Checkmarx Security Research (2023). Analysis of malicious PyPI packages matching ChatGPT hallucination patterns. Checkmarx Blog. Documented credential-stealing payloads in packages registered under AI-hallucinated names. ↩

5. npm Security Advisory (2023). Multiple reports of malicious packages with names matching common AI coding assistant hallucinations, including cryptominer payloads. npm Security. ↩

6. Lasso Security (2024). Research on AI model supply chain risks via hallucinated Hugging Face organization and model names. Lasso Security Blog. Demonstrated registration of matching repos as proof of concept. ↩