On July 16, 2026, Moonshot AI unveiled Kimi K3, a 2.8-trillion-parameter model that immediately topped a frontend-coding arena, with its open weights promised for July 27. This happens every few weeks now. A new frontier model lands, the benchmarks reshuffle, and every AI-native tool quietly decides whether to adopt it. For most tools a fresh model is a routine feature update. For a testing tool it should worry you, because it raises a question nobody markets the answer to: when the model changes, does my test suite change too?
If the honest answer is "yes," you don't own a test suite. You own a prompt that a model happens to interpret a certain way today.
The quiet failure mode of AI-at-run-time
Here's the architecture that creates the risk. Some AI testing tools store your test as natural language — "log in, add the backpack to the cart, check the badge shows 1" — and hand that sentence to an LLM every time the test runs. The model reads the page, decides what to click, decides whether the outcome counts as a pass, and moves on. It feels magical because it's flexible: the test "adapts" when your UI shifts.
But look at what you've actually built. The interpreter is now a dependency of every run, and it's a dependency you don't control and can't pin. Consider what changes underneath you without a single commit to your app or your tests:
- Silent model upgrades. The tool swaps its default model to the newest release (this week's K3, say) because it benchmarks better. Your test now runs through a different reasoner with different judgment about what "the badge shows 1" means.
- Deprecations. The model your suite was tuned against gets retired. The tool migrates you to a successor. Same test text, different behavior.
- Behavior drift. Even the "same" model, re-served or re-quantized, can decide differently on an ambiguous page. A borderline element that read as a match last month reads as a miss this month.
None of these touch your application. Yet any of them can turn a green test red, and any of them can do the far more dangerous thing and turn a red test green, the interpreter shrugging at a broken flow it has decided is "close enough." That's the case that should keep you up at night: a real regression that a slightly more lenient model waves through. You didn't relax the check. The model did, and nobody told you.
There's a simple diagnostic here. Can the same test, against the same build of your app, pass on Monday and fail on Thursday with no code change on either side? If model churn can move your bar like that, then what your suite reports on a given day depends partly on which model happened to be interpreting it, which makes it a reading of the interpreter rather than of your app.
A test is an artifact, not a conversation
The fix is architectural, and it comes down to when the AI runs.
TestVibe uses AI to write the test exactly once, then never again for that test. You describe the behavior in plain English, an agent explores your live app, and it produces a standard, deterministic Playwright spec: real code you can read, review, and check into version control. That generation step consumes AI and is metered as such, but the output is frozen the moment it lands. How generation works walks through the full pipeline. The load-bearing part for this discussion is the final step, where a generated test only reaches "generated" status after the complete spec has been replayed end-to-end against your real app and passed. Verified first, then delivered.
After that, runs have zero AI in the loop. When your nightly suite fires, it executes compiled Playwright against a browser. No model reads the page, and no model decides what "pass" means. The assertions you can see in the file do that, the same toHaveCount(0) and toBeVisible() matchers every time. This is why TestVibe's credit model meters generation on AI tokens but bills runs on cloud session time only: there are no AI tokens per run because there is no AI per run. You can read that split straight off an invoice, and it falls out directly from an architecture in which the model has already left the building by the time your test executes.
That gives you the line that matters. Model churn affects your next generation, never your existing suite. A frontier model drops, a provider deprecates an old one, the roster reshuffles, and every test you have already generated keeps running exactly as before, because it is Playwright code rather than a prompt awaiting interpretation. The determinism your CI leans on lives in the committed artifact, wherever the model that wrote it has since gone.
"But what if the model gets weaker?"
Fair question, and it has an honest answer instead of a reassuring one. A different or weaker model shows up as retries and generation time, never as a wrong-but-green test. Because every generated spec has to pass a real run against your live app before you ever see it, a model that reasons less well simply fails or stalls more often during generation. It costs you a slower first draft. It cannot ship you a test that looks passing but proves nothing, because a test that doesn't actually pass never gets marked generated in the first place.
Contrast that with the interpret-at-run-time model, where a weaker or more permissive interpreter degrades silently, and it does so on exactly the run you least want it to, the one you are trusting to gate a deploy. One architecture pushes model risk to the front, where it's visible and cheap. The other buries it in production, where it's invisible and expensive.
The ownership test: could you walk away tomorrow?
There's a second, colder way to check whether your tests are durable assets or rented behavior: if you cancelled the tool tomorrow, would the tests still run?
With a prompt-interpreter, no. The natural-language test is inert without the tool's model behind it. The behavior lived in the vendor's inference stack, and it leaves with the subscription.
TestVibe's answer is two-way Git sync. Your Gherkin features and their generated Playwright specs mirror into a repository you own on GitHub, GitLab, or Azure DevOps, laid out as a complete, runnable Playwright project with a package.json and a runner. Clone it and the specs run with the bundled runner, on your machine or in your CI, with no call back to TestVibe for the run itself. The tests are files in your repo. They diff in pull requests, they show up in git blame, and they outlive any decision you make about the platform. That isn't an escape hatch bolted on after the fact; it is what owning your test as an artifact actually costs the vendor to give you.
This durability compounds with everything else that makes behavior-level tests survive change. The same property that lets a spec survive a framework major upgrade, namely coupling to what the user does rather than to how the framework renders it, is what lets it survive model churn. The test never asked which model wrote it, the same way it never asked which framework version drew the button. Both of those are the vendor's problem to solve, not your suite's.
The line to hold
AI is genuinely useful for the hard, creative part of testing: turning intent into a working spec against a real app. That's worth paying for, and TestVibe meters it honestly. What AI should not be is a permanent runtime dependency sitting between your test and a verdict, re-litigating pass and fail on every execution while the model underneath it shifts release to release.
So use AI once, at authoring time, and keep the compiled result under version control where it runs with no model behind it. A new model landed this week and another will land next month; a suite built this way registers none of it. The only place the K3 question ever reaches is your next generation run, never the specs already sitting green in CI.
Want tests that are real Playwright code, verified against a live run, and yours to keep? Get early access, or read how generation actually works.