Pick a bad week and you'll find one: GitHub Actions hosted runners backed up in a queue, jobs that normally start in seconds instead waiting, timing out, or never running at all. If your entire test signal lives inside a .github/workflows file, that's the week you shipped blind. Your tests weren't broken. The machine that runs them just never showed up.
That's the failure worth thinking about, and it isn't specific to one provider or one bad day. Runner fleets degrade. Regions have incidents. Outage trackers log a steady drip of CI and platform disruptions across every major provider (ThousandEyes keeps a running timeline). The question isn't whether your CI will have a bad hour. It's what you lose when it does.
For most teams the answer is everything, and you can design your way out of that.
CI is a trigger, not a test runner
Here's the conflation at the root of it. "CI" does two jobs that we've quietly welded into one:
- It decides when tests should run: on push, on PR, on merge, on a schedule.
- It provides the machine the tests run on, the hosted runner that checks out your code, installs browsers, and executes.
Those are different concerns. The first is a scheduling problem, the second an execution problem, and when they live in the same system an execution outage takes out your scheduling too. You lose signal for reasons that have nothing to do with either your code or your tests.
Untangle them and the outage changes shape. If the machine that runs the test is independent of the thing that decided to run it, a CI brownout costs you one convenient trigger while every other path to the same test stays open. You degrade gracefully instead of going dark.
Decouple execution from the trigger
The move is to run tests somewhere that isn't your CI runner, and let anything fire them. Push execution into independent cloud sandboxes, a fresh isolated session per run spun up on demand, and the CI job shrinks down to an API call that says "run this suite now, tell me the result." The heavy lifting happens elsewhere. GitHub Actions being slow now means your notification is slow, not that your tests didn't run.
This is exactly the seam TestVibe is built around. Tests execute in isolated cloud sessions regardless of who asked. The CI/CD integration is a thin trigger: your pipeline kicks off a run and reads back pass/fail, but if the pipeline is wedged, the run doesn't depend on the pipeline's health to happen. Same suite, several front doors.
Once execution is decoupled, count how many independent ways you can fire the identical test:
- From CI, on push or PR. This is the default, and the one that just failed you.
- From a schedule, so a nightly or hourly run happens whether or not anyone pushed. A nightly regression playbook that runs on its own clock keeps producing signal straight through a daytime CI incident.
- From another run finishing, via an automation chained to fire when a prior run passes, so a fast smoke pass hands off to the full suite with no pipeline in the loop.
- From a CLI or API call, straight from a laptop or a script. The
testvibeCLI, REST ops API, and MCP server all kick off the same run against the same sandboxes. When Actions is down,testvibe runfrom your terminal is a thirty-second workaround, not a rebuild of your test infrastructure. - From a click, in the app, when you just want to confirm a thing before you merge by hand.
Five triggers, one execution path. Lose any one trigger and the other four still reach the same result.
The reverse problem: honest gates when the fleet is degraded
Decoupling also fixes the failure that's quieter and more dangerous than "no signal at all": false signal during a degraded window.
When a runner fleet is under stress, jobs don't only fail, they behave weirdly. They get slow. They hit undersized boxes. They time out mid-test. If your deploy gate reads "the CI job didn't come back green," a degraded fleet produces two bad outcomes at once. Real regressions slip through because the job never finished and someone force-merged past a "pending" check. And phantom failures block good code because a test that's fine flaked on an overloaded shared runner, the classic environment drift you get from running on whatever hardware the fleet happened to hand you this hour.
Isolated execution helps on both counts. Every run gets a clean, dedicated session instead of a contended shared runner, so a slow-fleet morning doesn't inject flake into your results. The sandbox your test runs in is the same whether the broader fleet is calm or on fire. And because the gate reads a real pass/fail from that run rather than the status of a CI job, you can keep gating on the answer even while the job orchestration around it is unreliable. Point the gate at the run result, not the runner's mood.
Two practical rules fall out of this:
- Gate on the test result, not the job's exit code. A CI job can be yellow for infrastructure reasons that say nothing about your code. Read the actual run outcome from the system that executed it.
- Keep at least one trigger that doesn't share fate with your primary CI. A schedule or a run-chained automation on a separate path means an Actions incident degrades convenience, not confidence.
The test for your setup
Ask one question: if my CI provider had a two-hour outage right now, would I still have test signal?
If the honest answer is no, your tests are coupled to your CI runner, and you're one bad status-page morning away from shipping blind. A second CI provider won't fix it, and neither will a pile of retry logic. Move execution off the runner entirely, and the runner becomes one trigger among several: still the convenient one, no longer the thing everything hangs on.
The concrete test is whether, during the next Actions incident, you can type testvibe run from your terminal and still get a real pass/fail in under a minute. If you can, the outage cost you a notification. If you can't, it cost you the whole afternoon.
Want tests that run in isolated cloud sessions you can trigger from CI, a schedule, a chained run, the CLI, or a click? Get early access, or read how the CI/CD integration wires a run into your pipeline as a thin trigger.