How to end-to-end test the AI features in your own app

You shipped an AI feature this year. A chatbot, a "summarize this thread" button, an ask-AI search box, a draft-reply assistant — something with a model behind it. And the first time QA sat down to write a test for it, they hit the wall everyone hits: the output is different every run, so toHaveText('...') is dead on arrival. The feature works, but nothing you'd normally assert holds still.

The models underneath keep getting better and cheaper — Moonshot's Kimi K3, announced July 16, is a 2.8-trillion-parameter model that currently leads Arena's front-end coding leaderboard — but a stronger model doesn't make the feature easier to test. It makes the output better, which is a different axis from testable. This is a solved problem, and the solution starts with refusing to test the one thing that's genuinely untestable in an e2e run.

The line that makes everything else work

Draw it before you write a single assertion: end-to-end tests own "the feature works." Evals own "the answer is good."

E2e can prove that a request goes out, a response comes back, it renders in the right container, the citations link somewhere, the retry button retries, and a rate-limited user sees the right message instead of a spinner that hangs forever. E2e cannot prove the summary is accurate, the tone is right, or the model didn't hallucinate a refund policy. That's a semantic quality question, and the honest tool for it is an eval harness with graded examples — not a Playwright assertion pretending a fuzzy answer is a fixed string.

Teams that ignore this line write one of two broken things: a golden-transcript test that fails every time the model rephrases, or a test so loose it passes when the feature is completely down. The techniques below all fall out of respecting the boundary — assert hard on the deterministic parts, assert loosely on stable invariants, and hand semantic quality to evals where it belongs.

1. Assert structure and behavior, not exact text

The response text is nondeterministic. The shape around it is not. When a user submits a prompt, a deterministic sequence still has to happen: a loading state appears, a response container renders, non-empty text streams into it, and any affordances — copy button, citations, thumbs-up, retry — show up. Every one of those is a rock-solid assertion that has nothing to do with what words the model chose.

await page.getByRole('textbox', { name: 'Ask a question' }).fill('What is your refund policy?');
await page.getByRole('button', { name: 'Send' }).click();

// The response container appears and fills — generous timeout for streaming
const answer = page.getByTestId('assistant-response');
await expect(answer).toBeVisible();
await expect(answer).not.toBeEmpty({ timeout: 30_000 });

// The affordances that prove the feature finished, not just started
await expect(page.getByRole('button', { name: 'Copy' })).toBeVisible();
await expect(answer.getByRole('link')).toHaveCount(2); // two citations rendered

The one place people flake here is streaming. A model reply can take fifteen seconds to finish, and a default five-second assertion times out mid-stream. Don't paper over it with a fixed sleep — raise the timeout on that specific assertion. This is exactly what auto-retrying web-first assertions are built for: they poll until the container fills or the ceiling is hit, so a slow generation costs you the extra seconds it actually needs and a fast one returns immediately. Never waitForTimeout a stream.

2. Pin the deterministic shell around the model

The model is one node in a flow that is otherwise completely conventional, and that shell is where most of the real bugs live. Auth still has to gate the feature. The input still has to validate and handle a 4,000-character paste. The error states — the model timed out, the provider 500'd, the user hit their quota — each need a real, tested UI path, not a blank panel. Rate-limit UX especially: what does the eleventh request in a minute show?

None of this is AI-specific, and all of it is fully deterministic. You describe these states in plain language and test them the same way you'd test any form and its unhappy paths — the empty submit, the oversized input, the server rejection, the recovery. A test that submits an empty prompt and asserts the Send button stays disabled is worth more than any attempt to grade the model's prose, because that bug ships silently and annoys every user.

// Quota exhaustion shows a real message, not a dead spinner
await expect(page.getByText('You've reached your daily limit')).toBeVisible();
await expect(page.getByRole('button', { name: 'Send' })).toBeDisabled();

3. Assert stable invariants, not golden transcripts

For the content itself, you can still assert — just not on the whole string. Good AI features have invariants that hold across every valid generation, and those make sound assertions. A product-support answer should mention the product name. A summary should be shorter than its input. A response should never contain your internal error strings, a raw stack trace, or the literal text undefined. A JSON-mode endpoint should return parseable JSON with the required keys.

const text = await answer.textContent();
expect(text).toContain('Acme'); // it's answering about our product
expect(text).not.toMatch(/undefined|NullReferenceException|\[object Object\]/);
expect(text.length).toBeLessThan(sourceText.length); // a summary, not an expansion

These are negative and structural checks — cheap, stable, and genuinely catching regressions like a prompt change that leaks system text or a broken template that renders undefined. What they deliberately don't do is judge whether the answer is good. That's the boundary again, and holding it is what keeps this test green for years instead of failing on every reword.

4. Mask the response region in visual baselines

Visual regression is worth running on an AI feature's page — the layout, the empty state, the composer, the citation chips all have a correct appearance you want pinned. The problem is the one region that changes every run: the response text itself will fail a pixel diff forever. So mask it. Playwright's native screenshot matcher takes a mask option that blacks out a locator before comparing, so the rest of the page stays under strict diff while the volatile region is ignored:

await expect(page).toHaveScreenshot('assistant-panel.png', {
  mask: [page.getByTestId('assistant-response')],
});

Now a z-index regression that drops a banner over the composer still fails the test, but a differently-worded answer doesn't. This is the same scoping discipline that keeps visual regression from drowning you in noise on timestamps and live counts — mask what moves on its own, pin everything else.

5. Seed determinism where the app allows it

The most durable fix is to make the feature less nondeterministic in test mode, if you own the code. A test-only flag that pins the model to temperature: 0, a fixed system-prompt mode, a canned-response fixture the endpoint returns when a test header is present — any of these collapses the fuzz and lets you assert harder. This isn't cheating: you're testing the plumbing (does the request go out, does the response render, does the citation link work) against a fixed input, which is precisely what e2e should own. Keep one small suite that hits the real model to catch integration drift, and run the rest against the deterministic seam.

The boundary, stated plainly

Don't let anyone talk you into grading answer quality inside your Playwright suite. A test that asserts "the summary is accurate" is either secretly loose or perpetually flaky — there's no third option, because accuracy isn't a fixed value. Quality belongs in evals: a graded dataset, run on a schedule, that scores model output and trends it over time. Your e2e suite proves the feature is wired correctly and fails loudly when the plumbing breaks. Two tools, two jobs. Conflating them gives you a suite that's confident about nothing and a quality signal you can't trust.

How TestVibe fits

You describe each of these behaviors in plain language — "submitting a question streams a non-empty answer with two citations and a working retry button," "hitting the daily limit disables Send and shows the quota message" — and TestVibe generates the Playwright test, then verifies it by actually running it against your live app before you ever see it. The generated spec is deterministic Playwright: generation uses AI, but the test that lands runs the same every time, with no model in the loop at run time. That's the right shape for testing an AI feature — a fixed, fast, honest check on the parts that must hold still, with semantic quality left to the evals where it belongs.

Get early access and describe your assistant's happy path and its three worst error states — that's a testable feature in about a paragraph.