Same Rows Every Run: Why CI Needs Deterministic Test Data

A test failed on the build server at 2am. By the time you look, the runner is gone, the database is gone, and the only artefact is a stack trace and a red tick.

You run it locally. It passes. You run it again on CI. It passes.

At that point the ticket gets a label like flaky or intermittent, and it enters the pile that nobody empties.

The problem is not that the failure was random. The problem is that you cannot get back to the conditions that produced it, so you cannot tell whether it was random or not.

The distinction that matters

Reproducibility is not about making tests pass. It is about making failures investigable.

A test that fails once in fifty runs is telling you something real. It might be a race condition. It might be an unhandled edge case in one generated value. It might be genuine environmental noise.

You cannot tell which, because you cannot run it again under the same conditions. So the information is lost, and the failure gets reclassified as noise, which is a decision made on no evidence.

Do that consistently for a year and your CI signal is worth very little. Everybody knows some failures are real and nobody knows which, so the rational response is to re-run the build, and that is where most teams end up.

Random data is right, unseeded random data is not

There is a good argument for randomised test data. Fixed fixtures only ever test the cases someone thought of. Randomised values find the ones nobody did: the apostrophe in a surname, the four byte emoji, the empty string that is not null, the number at the boundary.

The mistake is not randomness. The mistake is randomness you cannot replay.

If the generator is seeded and the seed is recorded, you get both properties at once. Each run explores different values, so you keep the discovery. Any specific run can be recreated exactly, so you keep the investigability.

This is standard practice in property based testing, and it is the single most useful idea that area has produced. When a property test fails, it prints the seed and the case. You paste the seed in and you are looking at the same failure. Nothing about that idea is specific to property testing.

What to record

Whatever generated the data, the run should record enough to rebuild it:

The seed. The generator version, because a change in generation logic changes the output for the same seed and that will confuse someone six months from now. The schema version or migration state, since the same generator against a different schema produces different data. The dataset itself, or a reference to it, if it is small enough to keep.

That is not a lot of metadata. It is the difference between "the suite failed" and "the suite failed on this exact data, here it is".

The workflow it enables

CI fails at 2am. The build output includes the seed.

In the morning you regenerate that dataset locally, run the one failing test, and watch it fail on your machine. You now have a normal debugging session instead of an archaeology project.

You fix it. You add that seed as a permanent regression case, so the specific data that broke you is now part of the suite forever.

None of this is sophisticated. It just requires that the data not evaporate when the container exits.

The honest limit

Determinism in data does not give you determinism overall.

Concurrency, wall clock time, network timing, and iteration order can all vary between runs with identical data. Fixing the data narrows the search dramatically, but it does not eliminate every source of variation, and anyone who tells you otherwise has not chased a race condition.

What it does is separate the two cases. If the data is pinned and the failure still comes and goes, you have learned something specific: the problem is timing, not input. That is a much smaller haystack.

Where SeedQL fits

SeedQL records its generation runs, so a dataset used by a build is identifiable after the fact rather than being a thing that existed briefly and then did not. Generations, snapshots, and pushes are tracked per project, with an audit trail.

I want to be careful about the claim. This is record keeping and reproducibility of the generation step. It is not a property based testing framework, it will not shrink a failing case to its minimal form, and it does not integrate with your test runner to automatically pin a seed on failure. Those are good ideas and they are not what is shipping today.

What it removes is the most common reason data based failures go uninvestigated, which is that nobody can say what the data was.

The question for your pipeline

Pick your last unexplained CI failure. Ask whether you could recreate the exact dataset it ran against.

If not, that failure was never going to be fixed, whatever anyone wrote on the ticket. And the next one will go the same way.