4 min read

The Data Your Bugs Live In Never Survives the Copy

Here is a pattern worth noticing.

Teams copy production data because they want realism. Then they sanitise it, because they cannot leave customer records lying around. The sanitising process normalises names, blanks free text, truncates long fields, drops columns that looked risky, and removes rows that failed validation.

What comes out the other end is clean, uniform, and well behaved. Which is to say it is missing every single thing that causes defects.

You went to production for the weird data and then removed the weirdness.

What actually breaks software

Almost no bug is caused by a typical row. Typical rows are what you developed against. The failures cluster around the edges:

The surname with an apostrophe, which breaks a query somebody built by concatenation. The name in a script your font does not have. The four byte emoji in a display name, which fits in utf8mb4 and not in utf8, a distinction MySQL has punished people for over many years.

The address field containing a newline, which corrupts a CSV export. The 300 character company name that a form allowed once, in 2016, before the validation was added. The NULL in a column your code assumes is always populated, because it was always populated when you checked.

The record created before a migration, which has a shape nothing else has. The account with 40,000 orders. The user whose email address has a plus sign in it. The row with a timestamp during a daylight saving transition, which exists twice or not at all depending on how you look at it.

These are not exotic. Every mature production database is full of them. And the sanitising script removes them, because from a sanitising point of view they look exactly like the problems it exists to solve.

The trade nobody notices

The team believes it made one trade: less privacy risk, in exchange for some effort.

The trade it actually made is bigger. It kept a copy of real customer data, with all the obligations that carries, and simultaneously destroyed the property that made the copy worth having.

That is the worst position of the available options. You are carrying the risk of holding real data and getting a dataset no more useful than a generated one, arguably less useful, because a generated one can be told to include the awkward cases on purpose.

That last part is the bit I find genuinely persuasive. Production gives you edge cases by accident, and only the ones that have happened to occur. Generation lets you ask for them.

Edge cases you can specify

If data is generated rather than harvested, the awkward cases become a choice rather than a lottery.

Unicode across scripts, including combining characters and right to left text. Names with apostrophes, hyphens, and spaces. Strings at exactly the column limit, and one under. Empty strings distinguished from nulls. Timestamps at midnight, at month boundaries, in February, on leap days. Numbers at zero, at one, and at the maximum the column type allows. Nulls in every nullable column, at a rate high enough that some code path actually meets one.

You can decide to include all of it, every run, deliberately. Production will give you some of that eventually, in an order you do not control, and only after a customer has found it first.

What generated data genuinely lacks

I do not want to oversell this, because there is a real gap and pretending otherwise would be the same dishonesty in the other direction.

Production data contains history. The account created in 2011 that has survived three schema migrations and carries a combination of values that no current code path can produce. The record written by a version of the application that no longer exists. The consequences of a bad deploy two years ago that were never fully cleaned up.

Those rows encode your system's actual past, and you cannot generate them, because generating them would require knowing about them. Nobody does. That is what makes them dangerous.

So the honest position is: generated data covers the edge cases you can name, plus volume and structure. It does not cover the archaeology. For most testing that is the right trade, and it is a far better trade than a sanitised dump, which gives you neither.

Where SeedQL sits

SeedQL generates from your schema, so the type and constraint boundaries are visible to it. It fills a relational schema at volume with the foreign keys intact.

Being precise about the current state: the value of generating rather than copying is that the awkward cases are something you can specify instead of something you hope survived a scrub. How much of that is configured by you versus inferred automatically depends on the shape of your schema, and I would rather you evaluate it against your own database than take a claim from a blog post.

The argument I actually want to make is the one above it. Copying production and then sanitising it is the option that costs the most and delivers the least, and it stays popular because each half of it is individually defensible.

Try this

Take the last five customer reported bugs that were data related. For each one, ask whether the triggering record would have survived your sanitising script.

If most of them would not have, your realistic dataset is not realistic. It is production data with the interesting parts filed off, and you are still carrying the risk of holding it.