This week it's Kimi K3. Moonshot AI shipped a 2.8-trillion-parameter model that topped a blind front-end coding arena over the leading US systems, with open weights promised to follow, and by next week there will be a new leaderboard with a new name at the top. The churn is now a fixed feature of the calendar. And every time it turns over, the same question lands in testing Slack channels: should we switch the model that writes our tests?
It's the wrong question. The trouble isn't that the answer is no. The trouble is the premise. The question takes for granted that correctness lives inside the model, and in a pipeline that's built right, correctness lives somewhere else entirely.
Why "which model" feels terrifying
The fear is rational if you've only ever used raw code generation. Ask a model, any model, to write a Playwright test for your checkout flow and it will produce a file that looks right. It'll have page.getByRole calls, some expect assertions, a plausible structure. It will also, with some nonzero probability, invent a selector for a button that doesn't exist, assert against a success message whose text it guessed, or wait on a network call your app never makes.
Every model hallucinates differently. A stronger model hallucinates less often and more subtly, which is arguably worse, because the failures are rarer and get trusted more. So when you choose a model for unverified generation you really are making a correctness decision under uncertainty, betting that this particular model's failure distribution is one your reviewers will actually catch, spec after spec, at the moment the file looks right at a glance and the sprint is already late. That is a scary bet. It's why "which model is smartest" feels like it matters so much.
It only matters that much because the pipeline is missing its most important stage.
The compiler already solved this
You don't trust GCC because you've read its source. You've never read its source. You trust it because you compile your program, run it, and the output does what you meant. The compiler is a black box whose correctness you verify by its output, not by inspecting its internals or checking which version topped a benchmark. If a new GCC release started producing subtly wrong binaries, your test suite would catch it. The verification lives downstream of the tool, not inside it.
Test generation should work the same way. Treat the model as the engine that turns your intent into an artifact and stop interrogating its pedigree. What makes the artifact trustworthy is a verification stage that runs after the model and gates on real behavior. That's the whole idea behind verified generation, and it's the reason you can trust AI-written tests at all: the trust doesn't come from the model, it comes from the check that the model's output has to survive.
What verification actually gates on
In TestVibe, a generated spec doesn't reach your project as a test until it has passed a real run against your live app in an isolated cloud session. Passed, not compiled and not merely plausible-looking: the complete assembled spec, replayed end-to-end from a cold browser, the same way the pipeline is built to prove every generation before you see it. If the agent invented a selector, the run fails and the agent tries again. If it asserted against a message that never appears, the run fails. If it waited on a phantom network call, the run fails. None of those failures reach you as a green checkmark. They reach the agent as a retry.
Walk through what that catches, by failure class:
- Hallucinated selectors. A button the model imagined isn't on the page. The
clickthrows, verification fails, no test is produced. A stronger model imagines fewer of these; a weaker one imagines more. Neither can ship one, because the live page is the arbiter. - Invented assertions. The model guessed the success banner says "Order confirmed" when your app says "Thanks — your order's in." The assertion fails against the real DOM. Verification catches it regardless of how confident the model was.
- Fabricated waits and flow. A step that depends on state the app never reaches simply won't reproduce in a cold run. The verification stage runs from the entrypoint, so anything that only worked in the agent's warm, already-navigated session gets exposed before delivery, not in your nightly suite three weeks later.
None of these safeguards are model-specific. They're the same gate for every model, because the gate is your live app rather than the model's own say-so.
Benchmarks predict speed, not correctness
This reframes the whole "which model" conversation. Benchmark rankings (the arena scores, the agentic-eval leaderboards, the number that made Kimi K3 news) are real signal, but they predict the wrong variable for your peace of mind.
A model that's genuinely better at front-end coding will, inside a verified pipeline, hallucinate fewer selectors and get scenarios right in fewer attempts. That shows up as fewer retries and faster generation: less time in the drawer, less compute burned per feature, more first-attempt successes. It does not show up as more-correct tests, because the tests that reach you are already correct in both cases. The floor is the same. What moves is how expensively you got there.
This is the honest answer to "what if we're stuck on a weaker model?" A weaker model costs you retries and wall-clock time during generation. It cannot cost you a wrong-but-green test, because a spec that doesn't actually pass a live run never earns the generated status. The failure mode of a weak model is "more incomplete generations," never "silently broken suite." That's a cost you can see and budget for, not a risk hiding in your CI.
It also changes how a model swap feels. Upgrading to a stronger model doesn't force you to re-audit anything, because your suite was already correct; you're buying the same trustworthy output for less compute. Model choice ends up being a dial on speed and cost, one you can turn without touching correctness at all.
Grading against reality, not against itself
There's a subtle trap that verified generation specifically avoids: letting the agent be its own judge. An agent that writes a test and then reasons "yes, this looks correct" is grading its own homework, and a confident model grades its own hallucinations as passing. The whole value of verification is that the grader is not the model. It's a real browser hitting your real app, returning a real pass or fail. No amount of model confidence overrides a failed run.
This is why the pattern survives the model roster changing underneath it. On cloud, the engine is TestVibe's first-party service running frontier models. On a self-hosted install you can point generation at your own OpenAI-compatible endpoint, a model server inside your own network, and the verification stage still runs, unchanged and un-skippable. A weaker on-prem model produces more failed generations and zero false-green tests, exactly like the cloud case. The gate doesn't care whose weights are behind it.
The durable part
Models are the fastest-moving thing in this entire stack. Whatever tops the arena today will be mid-tier in a quarter. Building your test suite's correctness on top of a specific model's smarts means re-litigating trust every release cycle.
Verification is the part that doesn't move. A generated, verified Playwright spec is deterministic — it consumes AI to be created, but its runs use no model at all. Once it's proven against your live app, it keeps running identically no matter what happens to any leaderboard. The model that wrote it can be deprecated, replaced, or beaten by an open-weight upstart from Moonshot, and your suite doesn't notice.
The durable question here has nothing to do with which model writes the best tests. What matters is whether your pipeline checks its own output against the running app before it calls a spec done. Settle that once and the model roster underneath you can turn over as fast as Moonshot ships.
See it on your own app
Verified generation is the whole point of TestVibe: every generated test passes a real run against your live application before it reaches you, whichever model is behind the curtain. Get early access and watch a spec get written, verified, and marked green, then swap the model behind it and watch nothing about your suite's correctness change.