That AI Tool That 'Failed' Six Months Ago? It Might Just Have Needed More Time
The UK's AI Security Institute found that the standard way we test AI agents quietly undersells them — because we cut them off too early. Give them more computing time and they solve much more. A plain-language look at why this matters for anyone judging AI tools.
Here’s a finding that quietly reframes a lot of AI headlines. The UK’s AI Security Institute (a government body that studies AI safety) says the way we measure AI “agents” — AI that works through multi-step tasks on its own — is systematically unfair to them. Not because the tests are rigged, but because we cut the AI off before it’s finished thinking.
The core idea is intuitive once you see it. An agent gets better the longer it’s allowed to work — more attempts, more checking, more reasoning. Most tests, though, impose a budget: a limit on how much computing (measured in tokens, the chunks of text an AI processes) the agent may use. If you stop it while it’s still improving, your score reflects the floor of its ability, not the ceiling. The researchers found that on software-fixing tasks, success rates jumped about 25% when they raised the budget from one million to ten million tokens. Some security tasks were only cracked with 50 million. Same model, same problem — just given room to keep going.
There’s a second, more striking part. Because token budgets kept rising, the Institute’s estimate of how fast AI’s hacking-related skills are improving turned out to be too slow. Measured with generous budgets, the capability is doubling roughly every 40–50 days rather than every few months — meaningfully faster. That’s a safety-relevant finding, and a reminder that how you measure something shapes what you conclude. To be fair, more computing time isn’t a magic wand: on medical question tests, models hit a wall no budget could push past, and on some tasks newer models did worse. Extra time helps most where the AI can check its own work — like running code and seeing if it passes.
What this means for you: If you tried a local AI model or an agent tool a while back and it flopped, don’t write it off from memory — the tool may have been fine and the leash too short. And since the price of AI computing keeps falling, capability that felt too slow or too expensive last year quietly becomes practical this year. The concrete habit: when you give an AI a genuinely hard task — especially one with a built-in check, like tests or a compiler — let it run longer before you judge it. Patience is now part of using these tools well.
Sources
A Smaller, Custom AI Beat GPT and Claude at a Real Job — for a Fraction of the Cost
A giant hedge fund and a top AI startup showed that a free, downloadable model — trained on their own experts' judgment — outperformed every big-name AI on finance tasks, at roughly one-fourteenth the cost. The lesson applies to far more than Wall Street.
