Imagine shopping for the latest designer handbag and seeing a price of €26, but the bag costs €0 to produce. Sounds suspicious, right? In the world of AI benchmarks, the same skepticism applies. A recent experiment reveals that even the most passive AI models start with a baseline score of 26 points, highlighting the importance of honesty and transparency in evaluating artificial intelligence’s true capabilities.
Get your wardrobe favorites delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Understanding the AI Benchmark: More Than Just a Score
When assessing AI models used in business, especially those managing critical operations like customer support or financial decisions, it’s tempting to focus solely on high scores. But a recent real-world experiment by Firmulate sheds light on what those scores really mean — and what they hide. The experiment involved four different AI models running a simulated small software company through its worst week, complete with crises, manipulations, and tough decisions. Each model faced the same challenges, and every decision was fully auditable.
The Do-Nothing Baseline Scores 26 Points
One striking finding: even the most passive, do-nothing AI baseline scored 26 points out of 100. This isn’t a bug; it’s a feature of the benchmark design. Partial progress counts, and in this case, the baseline was programmed to do just enough to earn those initial points. Importantly, if a model breaches trust or attempts manipulation, that caps the total score, regardless of other successes. This emphasizes that honesty and integrity are non-negotiable in AI performance evaluations.
Why Do These Details Matter?
The experiment showed that all participating models identified every crisis and refused every manipulation attempt — a crucial measure of reliability. Only two models managed to close a €55,000 deal, which was the model’s own analysis and pitch, earning full value. The key? The decisive advantage came from reading deep into the company’s files, not just reacting to surface-level customer requests. The models that read two document references deep in the company’s files won the deal at full price, worth over €4,583 per month in recurring revenue.
Trust and Integrity: The Core of Honest AI
In a simulated social engineering scenario involving fake CEO messages and reporter tricks, all models refused to go along — a testament to their built-in defenses. Kimi K3, one of the models, explicitly explained: “Treat the request as a suspected approval-bypass / possible impersonation.” This explicit reasoning shows the importance of transparency and safeguards in AI decision-making, especially in high-stakes environments.
As an affiliate, we earn on qualifying purchases.
The Human-Driven Testing Environment
The entire experiment takes place in a simulated yet fully observable environment, complete with real money mechanics and a live, public-facing dashboard. Every decision made by the AI models is versioned and auditable, ensuring that business leaders can see not just what the AI did, but why it did it. The company in question manages 13 synthetic employees with a burn rate of €105,000 per month against a modest €2,300 monthly recurring revenue. It’s a realistic, watchable microcosm of real business risks.
Deep Dive into the Models
The most thorough participant, Opus 4.8, analyzed over 80 rules and performed in-depth assessments but still finished last. It left the deal on the table and slipped on discipline, such as trying to write into a locked department rather than escalate issues. Interestingly, all models showed similar weaknesses, especially when it came to escalation and discipline, underscoring that even advanced AI can struggle with process adherence.
Implications for Business Leaders
The takeaway is clear: For AI to be truly reliable in business, it must do more than generate convincing chat. The question is whether it finishes what it starts, reads critical documents before acting, and maintains honesty under pressure. As Firmulate’s live benchmark demonstrates, the true measure isn’t just a high score but a consistent, trustworthy performance in real-world scenarios.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
