Should you trust a test an AI wrote?

Short answer: no, you shouldn't take it on faith. But that's true of any test you didn't write yourself. A test checking behavior nobody verified is a liability whether a human or a model produced it, and asking whether AI in particular can be trusted to write tests is asking the wrong question. The one that matters is what has to be true before any test earns your trust, and the answer doesn't change with the author. It has to have run, and someone has to have read what it claims.

That standard is actually good news for generated tests, because a model can run its own output in a loop far faster than you can, catching its own broken selectors before you ever open the file. That execution is what earns the trust.

Why unrun code is the real risk

A test is a claim: "when the user does X, the app does Y." Generated code makes that claim in fluent, confident Playwright, and the fluency is what gets people into trouble. Because it looks right, you're tempted to skim it, nod at some reasonable-looking selectors and assertions, and commit.

Here are the failure modes that survive a skim but die on first run:

• Selectors for elements that don't exist. The model inferred a data-testid or a button label from context that isn't in your DOM. The test never reaches its assertion.
• Assertions that can't fail. The classic is a test that clicks through a flow and then checks something trivially true, so it passes no matter what the app did.
• Tests that navigate but never verify. Lots of page.click(), page.fill(), and page.goto(), and no expect() that pins the actual outcome. Green, and meaningless.
• Waits tuned to a lucky run. A hard-coded waitForTimeout that happened to be long enough once, and will flake in CI under load.

A tautological assertion is the most dangerous because it stays green forever:

// Looks like a login test. Proves nothing.
test('user can log in', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('user@example.com');
  await page.getByLabel('Password').fill('correct-horse');
  await page.getByRole('button', { name: 'Sign in' }).click();
  expect(page.url()).toBeTruthy(); // a URL always exists
});

That test passes against a completely broken login. The fix isn't more AI, it's an assertion tied to an observable outcome:

  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
  await expect(page).toHaveURL(/\/dashboard/);

None of these bugs are exotic. They're the normal failure surface of code generated from an intent rather than observed against a running system, and running it is how you tell a real test apart from a plausible-looking one.

Verification is what converts a draft into a test

The move that makes a generated test trustworthy is boring and mechanical: run it against the real application and see what happens. A selector that doesn't exist throws. An assertion that can't fail also can't tell you the app is broken, so you check it against a known-good and known-bad state. A flow that never asserts produces a green run that a reviewer can spot as suspiciously fast and empty.

This is where a generation system that closes the loop earns its keep. In TestVibe, generation doesn't just emit code. The agent explores the target site, writes the Playwright for each scenario, and then replays what it wrote against the live site. A failed replay loops back to writing and tries again, so a scenario that references a button that isn't there gets caught and rewritten before you ever see it. After every scenario is recorded, the whole assembled spec is run end-to-end one more time. As the docs put it, "Recording every step is not the same as the finished test passing a real run — this is the check that proves it does." A feature only reaches generated status once that final verification passes, and there's no way to skip it.

That's the honest version of "AI writes your tests": the output was executed against your actual app before anyone handed it to you, rather than merely looking like it would work. When a feature shows green here, a real browser drove that flow on the live site and the assertion held.

The human checkpoint is the spec, not the code

Execution catches code that doesn't run. It cannot catch code that runs and verifies the wrong thing. A test can pass perfectly while asserting a behavior you never wanted. That gap is where a human belongs, and the cheapest place to stand is the spec.

Read the Gherkin first. If the feature description is wrong, the generated code will faithfully, verifiably test the wrong behavior. TestVibe's own guidance is blunt about this: AI generation does not replace review, and you still decide whether the spec describes the right behavior, whether the generated code checks the right outcome, and whether a failure means the app is broken or the test is wrong.

A quick spec-and-code pass, before you lean on the test in CI:

• Does the scenario describe behavior you actually care about? A verified test of an irrelevant path is verified waste.
• Is there a real assertion, and does it pin the outcome? Look for an expect on visible state, not on the fact that a page loaded.
• Does the code follow the journey the spec describes? It's the same story told twice, and the Gherkin and the code should agree.
• Is the test data repeatable and free of accidental private data? Credentials belong in configured secrets, not baked into the spec.

You don't need to understand every line to catch the obvious wrong turns. You're not re-deriving the test. You're confirming the intent is right, and letting execution confirm the mechanics.

So, should you trust it?

Trust the ones that ran green against your real app and whose spec you read and agreed with. Distrust the ones that got merged because they looked reasonable, and apply that exact bar to your hand-written tests too, since an assertion nobody ran is worthless no matter who typed it.

When a machine writes the tests, the trust bar holds steady and your attention shifts: less time hand-cranking selectors and waits, more time reading intent, while the tedious, essential job of running each scenario until it actually passes happens without you. If you want to see the verification step in detail, the docs walk through how verification works; if you'd rather just try it, get early access to generation that has to run before it ships.