The million-token context window just went from frontier flex to table stakes. Moonshot AI's Kimi K3, released July 16, 2026, is a 2.8-trillion-parameter open Mixture-of-Experts model with a 1-million-token context window — the largest open-weight model shipped so far, though its weights aren't slated to land until July 27 and aren't downloadable yet. The pitch that comes bundled with every big-context release is seductive: the model can just see everything now. Your whole codebase. Your whole app. No more chunking, no more summarizing, no more losing the thread.
That's true, and it matters for test generation. But "the model can see it" and "the model got it right" are different claims, and the gap between them is exactly where testing lives. So it's worth being precise about what a huge context window actually buys you when an AI writes tests — and what it quietly does not.
What long context genuinely improves
Generating a good end-to-end test is not a one-shot code completion. It's an exploration problem. To write a test for "an admin invites a teammate and sees a success message," an agent has to open the real app, find the real invite button, watch what the real form does, notice that the success toast takes a beat to appear, and only then commit to code. That produces a lot of working state: page snapshots, accessibility trees, prior test files it should stay consistent with, the DOM as it actually rendered, and — when a run fails — a full trace of what happened.
Before million-token windows, all of that had to be compressed to fit. And compression is lossy in exactly the wrong way for testing. Summarize a page snapshot and you throw away the precise label text and role the locator needs. Summarize the last three tests in the suite and the model reinvents a login helper that already exists two files over. Summarize a failed run's trace and you lose the specific network timing that explains the flake.
A large context window lets the agent hold the whole working set at once, uncompressed. Concretely, that helps in three ways:
- Cross-page consistency. When the agent can keep every page it explored in view simultaneously, the login flow it writes for scenario one matches the login flow in scenario five. It stops contradicting itself across a suite.
- Pattern reuse. With the existing test files actually in context — not a summary of them — the agent reuses your naming, your fixtures, your locator conventions instead of generating a fourth slightly-different way to select the same button.
- Whole-trace debugging. When a generated test fails verification, the entire trace of that failed run can sit in context while the model reasons about the fix, rather than a lossy digest of it. This is the same reason a full Playwright trace beats a stack trace for humans: the signal is in the detail, and detail is what compression eats first.
None of this is hypothetical. It's the difference between an agent that treats your app as a fresh puzzle every scenario and one that builds a coherent mental model and keeps it. Bigger context makes the draft better.
What it emphatically does not fix
Here's where the marketing and the reality diverge. Three things a million-token window does not give you.
Context is not ground truth. A model can have your entire rendered DOM in its context and still write a selector that misses. It sees <button> elements and class names and text nodes; it does not know which one the user actually clicks, whether that class is stable or churns every deploy, or whether the visible "Submit" becomes "Place order" next sprint. Handing the model more of the page doesn't resolve the ambiguity — it just gives it more rope. The discipline that produces locators that survive a refactor — role and accessible name first, test IDs second, raw text last — is a strategy, not a byproduct of raw visibility. A bigger window shows the model more selectors to pick badly from; it doesn't make the pick correct.
Attention degrades across long contexts. "Fits in the window" is not the same as "attended to evenly." The well-documented failure mode of long-context models is that recall sags in the middle — a detail on page 30 of the working set gets less weight than the same detail on page 1 or page 60. So the promise that the model reliably uses everything you stuffed in is softer than the token count suggests. Loading a whole app's exploration into context improves the odds the relevant fact is present. It does not guarantee the model leans on it at the moment it writes the assertion.
No window substitutes for running the test. This is the one that matters most, and no amount of context closes it. A test is a claim about behavior — "clicking this shows that." The only way to know the claim is true is to execute it against the live app and watch. A model reasoning over a perfect, complete, uncompressed snapshot of your application is still reasoning over a representation of it: a moment frozen in tokens. The app has timing, races, state, a backend that returns real data. You cannot read correctness out of a context window; you can only observe it from a run.
Bigger context, better drafts. Verification, correctness.
Put those together and you get the actual division of labor. A larger context window raises the ceiling on how good the first draft is — more consistent, more grounded in your real patterns, better at using what it explored. That's genuine value, and models like K3 — which developers preferred over leading systems for front-end coding in blind Arena testing — will keep pushing that ceiling up.
But the draft is not the deliverable. Correctness comes from a different mechanism entirely: running the generated test against the live app before anyone trusts it. In TestVibe, a generated spec doesn't reach "generated" status until the whole assembled test replays end-to-end, cold, in a fresh isolated session, and passes. That gate doesn't care how many tokens the model saw. A weaker or cheaper model with a smaller window costs you retries and generation time — more attempts to get to a passing draft — never a wrong-but-green test that sails through. The verification step is what makes an AI-written test trustworthy, and it's model-agnostic by design: the same green bar means the same thing whether the draft came from a frontier model or a modest one.
This is also why generated tests are just deterministic Playwright. Generation spends AI; the resulting spec does not. Once a test is verified, it re-runs identically forever, regardless of what happens to any model, any context window, any pricing sheet.
The question to ask a vendor
If you're evaluating an AI-testing tool in a market where every release leads with its context window, the token count is the wrong headline number to fixate on. Two questions cut deeper:
- What does the model actually see? Real page snapshots and accessibility trees from your live app, prior tests, and full traces of failed runs — or a summary, a URL, and a hope? Context size only matters if the tool fills it with real observations.
- What gates the output? Does a generated test have to pass a real run against the live app before it's handed to you, or does it ship on vibes because the code compiled and looked plausible? Pricing tells you the cost per token — around $3 per million input and $15 per million output for K3 — but it tells you nothing about whether the test is correct. Only a run does that.
A million tokens is a real upgrade to how much an agent can hold in its head while it drafts. Treat it as exactly that — a better draft — and keep correctness where it belongs: in the run.
See verified generation on your own app
TestVibe drafts tests from a live exploration of your real application, then proves each one with a cold end-to-end run before you ever see "generated." Bigger models make the drafts better; the verification gate is what makes them trustworthy. Get early access and point it at a flow you care about.