Every AI testing tool has a question its landing page skips: when the model writes a test for your app, what leaves your network, and where does it land? For a public marketing site nobody cares. For an internal admin console, a healthcare portal, a government system, or anything covered by a data-processing addendum, that question is a procurement gate — the security review either clears it or the deal stops.
This is the part of "AI-generated tests" that engineers wave through and their compliance team does not. So let's be precise about what an AI test generator actually sends off-box, then map the honest spectrum of where you can make it run.
What actually leaves when a model writes a test
To generate a real test, the model can't work from your one-sentence description alone. It has to see the app. In practice that means some combination of:
- The URL under test — often a staging or internal hostname that itself reveals infrastructure.
- The rendered DOM — your markup, element labels, form fields, and the copy on the page.
- Screenshots — literal pixels of the page, including whatever data is on screen at that moment.
- Your description of the flow — which frequently encodes business logic ("an admin approves a wire transfer over the daily limit").
- The generated code and its run artifacts — traces, videos, logs.
None of that is exotic — it's the necessary raw material for grounding a test in your real UI instead of guessing at selectors. A model has to explore your live app before it can write a line of code. But "necessary" and "allowed to leave the building" are different questions, and the second one is answered by your data-residency posture, not your enthusiasm for AI.
The reassuring news is that where this material goes is a choice, not a fixed property of the tool. The tradeoff is operational weight against perimeter control, and it lives on a three-point spectrum.
Point one: cloud, first-party metered AI
The default. You sign in, describe a feature, and generation runs on TestVibe's cloud. The AI is a first-party metered service — there are no model-provider keys to bring and no third party in the loop that you have to separately vet. Generation runs against your app inside an isolated per-run sandbox: a fresh browser context that's created for that run and torn down after, so one run's state never bleeds into the next.
Credentials are the sharp edge here, and they're handled deliberately. Login passwords and API tokens live in a per-project encrypted store and are referenced from your spec as tokens like {{secret:TEST_USER_PASSWORD}}, never as literal values. The plaintext is injected only at run time inside the sandbox — after code generation has finished. The model that writes your test sees the name of a secret so the code reads the right environment variable; it never sees the value. That means the credential doesn't reach the generation context, the generated code, or the run logs.
For most teams — public apps, ordinary staging environments, no residency clause — this is the right answer, and worrying about model geography is worrying about the wrong thing. The cloud is genuinely the least to operate.
Point two: private runner (a cloud feature, not self-hosting)
Before jumping to a full on-prem install, know the middle option, because it solves a narrower problem far more cheaply. If your only constraint is the app under test must not be reachable from the public internet — an internal console, a VPN-only staging box, a build on your laptop — you don't need to self-host anything. You run the hosted app as normal and route execution through a tunnel or a private runner inside your firewall. The app URL stays unchanged, no inbound ports open, and traffic to your app stays on your side.
This is a data-reachability answer, not a full data-residency one: with a tunnel, generation still happens in TestVibe's cloud. Reach for it when the gate is "our app can't be public," and reach past it when the gate is "the control plane itself can't be hosted."
Point three: self-hosted, and self-hosted with your own model
When the requirement is that the whole platform runs on your infrastructure — air-gapped deployments, strict residency rules, a compliance regime that won't allow a hosted control plane — the self-hosted edition puts every service inside your network. It's a Docker bundle: the app, Postgres holding your projects and test files, an object store for artifacts, the runner, and a browser host. Your DOM, screenshots, traces, and canonical test files live in volumes you own. Billing and metering are off entirely; there are no credits and nothing is counted.
That still leaves one question: where does the model run? Self-hosted gives you two answers. Supply a hosted Anthropic or OpenAI key for the simplest setup and strongest quality — at the cost of generation prompts leaving your network for that provider. Or point generation at your own OpenAI-compatible endpoint — an internal gateway, a vLLM or Ollama server, or an inference sidecar baked into the bundle — and nothing leaves your perimeter at all. For an air-gapped install, the endpoint path is the only one that qualifies.
There's one hard requirement on that model: it must support tool/function calling, because generation drives a browser by having the model call navigate, click, and assert step by step. A model that can't call tools stalls rather than fails cleanly. Plan on a GPU, and know that the model's quality caps the quality of the tests it produces.
Here's the part that changed recently. "Keep prompts in your network" used to mean accepting a visibly weaker local model. It doesn't anymore. Moonshot AI's Kimi K3, announced July 16, 2026 with open weights promised by July 27, is a 2.8-trillion-parameter model that topped Arena.ai's frontend-code arena in blind testing. Open-weight models reaching that tier is what makes the private-endpoint path a real engineering choice instead of a compliance-driven downgrade.
The safety net doesn't move
Wherever the model runs, one guarantee holds: a generated test only reaches generated status after TestVibe replays the whole assembled spec end-to-end against your live app and it passes. This validation can't be turned off. So a weaker or private model can't silently ship a broken-but-green test — the worst it can do is cost you more failed or incomplete generations. And once a spec is generated and verified, it's deterministic Playwright: runs consume no AI at all. A verified test keeps executing identically no matter what happens to any model, in any datacenter. Your residency exposure is scoped to generation time, not to every run forever.
The buyer's checklist
Strip the marketing off any AI testing tool and ask four questions:
- What leaves? DOM, screenshots, URLs, flow descriptions, generated code — assume all of it during generation unless the vendor says otherwise.
- Where is it stored, and for how long? Artifacts and test files: whose disk, whose region, what retention?
- Who can read it? First-party service, or a chain of third-party model providers you now have to add to your DPA?
- What's the private-perimeter path? Tunnel for reachability, self-hosted for the control plane, custom endpoint for prompts that never leave — and which one your regime actually requires.
The right answer isn't "always on-prem." It's matching the point on the spectrum to the gate you actually have to clear, and knowing that with open weights at frontier level, the strictest point no longer costs you the best model.
See where your data would go
Map your own requirement before you buy: reachability, residency, or full air-gap. Get early access to try the cloud edition, or read the self-hosting guide and the custom-endpoint walkthrough to see exactly what stays inside your network.