Your Tests Aren't Flaky. Your Test Data Is.

There is a particular kind of meeting where someone says the test suite is flaky, and everyone agrees, and nothing happens.

Flaky has become a catch all. It means "fails sometimes and I do not want to look into it". And because the word implies randomness, it also implies there is nothing to find. Cosmic rays. Timing. CI being CI.

Most of the time that is wrong. Most of the time the suite is perfectly deterministic and the data underneath it is not.

The shape of the bug

Here is the usual version.

You have a shared test database. It gets seeded once, at the start of the run, from a fixture file. Then two hundred tests run against it.

Test A creates a user and does not clean up. Test B counts users and asserts the count is 5. Test B passes when it runs before A and fails when it runs after. If your runner randomises order, or shards across workers, or just changes order when someone renames a file, you get a failure that looks random and is not.

Then the fix goes in. Someone changes assert count == 5 to assert count >= 5. The test goes green. It also stops testing anything.

I have seen suites where a third of the assertions had been softened this way over a couple of years. Every one of them was green. Collectively they verified almost nothing.

Why retries make it worse

The other common response is a retry decorator. Failed? Run it again. Green on the second attempt? Ship it.

Retries are reasonable for genuine external flakiness, like a network call to something you do not control. They are terrible for data problems, because the second run often starts from the state the first run left behind, which is a different starting condition entirely.

So you get a test that fails on a clean database and passes on a dirty one. That is not a fix. That is a test whose meaning depends on how many times you ran it.

The tell is simple. If a test passes on retry more often than it passes on first attempt, you do not have flakiness. You have a fixture that depends on residue.

Three things that actually help

Make the starting state explicit. Every test should either create what it needs or run against a known snapshot. "The database happens to contain some users" is not a precondition, it is a hope.

Stop asserting on global counts. SELECT count(*) FROM users is a shared resource. Scope your assertions to rows your test created. If you cannot identify those rows, that is the actual problem to fix.

Make it reproducible. If the data is generated, seed the generator and record the seed. A failure you can reproduce on demand is a bug. A failure you cannot is a ticket that gets closed as "could not reproduce" six weeks later.

That last one matters more than the other two combined. The reason data bugs survive so long is not that they are hard. It is that by the time you look, the evidence is gone.

The reset cost problem

The honest objection to all this is time.

The clean answer is to rebuild the database between tests. Nobody does it, because restoring a realistic dataset takes long enough that a fast suite becomes a slow one, and a slow suite stops being run.

So people compromise. Truncate a few tables. Wrap tests in transactions and roll back, which works until something commits or you need to test across connections. Share the database and hope.

That compromise is where the residue comes from. It is not laziness, it is a reasonable response to a real constraint.

Which is why the useful version of this is not "reset more often". It is "make the reset cheap enough that resetting is the default".

Where SeedQL fits

SeedQL generates a dataset from your schema and can push it into a target database, so rebuilding a known starting state is a command rather than an afternoon. Runs are recorded, so when a suite fails against a particular dataset you can see what was generated rather than guessing.

I want to be accurate about the boundary here. SeedQL does not run your tests, does not know which of them are order dependent, and will not tell you that test B depends on test A. That analysis is still yours.

What it removes is the excuse. When getting back to a known state is expensive, sharing state is the rational choice and residue is inevitable. When it is cheap, you can stop building your suite around a constraint you no longer have.

The reframe

Next time a test fails and someone says flaky, ask one question: what was in the database when it ran?

If nobody can answer, that is your finding. The test is not unreliable. The test is fine. It is being run against a different world each time, and no assertion survives that.