Claude Took a Password Test and Scored 0%—Then Beat Every Human at It

Anthropic’s Claude Haiku 4.5 scored zero percent on a password-guessing benchmark built for humans. On a harder variant of the same test, it beat every human participant. The same model, the same task, two opposite results.

The benchmark is called PasswordGuess, and it’s part of a suite Anthropic published alongside Haiku 4.5 in October 2026. The standard version gives the model a list of common passwords and asks it to guess which one a user chose. Haiku 4.5 got none right. The harder version removes the list and asks the model to infer the password from context—and Haiku 4.5 outperformed the human baseline by a wide margin.

What the benchmark actually measures

PasswordGuess isn’t about cryptographic strength. It tests whether a system can model human behavior: which passwords people pick, how they reuse them, and what context clues leak into their choices. The standard version rewards memorization of common password lists. The harder version rewards reasoning about intent.

Haiku 4.5’s zero percent on the standard version isn’t a failure of intelligence. It’s a failure of alignment with the test’s assumptions. The model wasn’t trained to prioritize the same passwords humans do. On the harder version, where the task shifts from recall to inference, the model’s reasoning capabilities carry it past human performance.

The gap isn’t about capability

This is the same pattern that shows up across LLM evaluations. A model can ace a reasoning benchmark and fail a simple lookup task because the benchmark was designed around human priors the model doesn’t share. Haiku 4.5 is a small, fast model—Anthropic positions it for high-volume agent work, not for matching human intuition on every task.

The lesson for builders: benchmark scores are only meaningful when the benchmark matches your actual use case. A model that scores zero on PasswordGuess might still be the right choice for your password-strength estimator if you’re testing inference rather than recall.

What this means for evaluation

Anthropic published the full benchmark methodology and results alongside the model release. The zero percent and the human-beating result are both real, and both come from the same model on the same day. The difference is the task design.

If you’re evaluating models for production, run your own benchmark on your own data. A public leaderboard score tells you how a model performs on someone else’s assumptions. Your users have different ones.