Catching data errors · Clarke Systems Group

Most data errors are not hard to spot. A missing size attribute, a price that differs from the retailer feed, a product with two images where four are required. Any competent reviewer would catch each one. The problem is that nobody can catch all of them, every day, across thousands of records. This guide explains how to replace that reliance on attention with a system of automated checks that runs before anything is released.

Why manual review fails at volume

Manual review works when the volume is small and the rules are few. Neither stays true for long. As your catalogue grows across markets, retailers and channels, the number of things that can go wrong multiplies.

Reviewers get tired. They check the fields they expect to be wrong and skim the rest. Two people apply the same rule differently. Under deadline, review is the step that gets shortened. And when an error does reach a live listing, there is rarely a record of what was checked, by whom, or against which rule.

The failure is structural, not personal. A person can apply judgement. A system can apply the same rule to every record, every time, and write down the result. You need both, but the second should come first.

Designing validation rules

A validation rule is a precise, testable statement about what a correct record looks like. If you cannot express it as pass or fail, it is not yet a rule. Most useful rules fall into five types.

Completeness

Are the required fields present? A product record for the German market may need a size, a colour, a material and an EAN. If any is blank, the record fails. Define required fields per market or channel, because requirements differ.

Format

Is each value in the expected shape? Barcodes have a fixed length and check digit. Prices use the right decimal separator. Titles sit within the retailer's character limit. Image URLs resolve.

Consistency against a source of truth

Does the value match the system that owns it? The price in your listing should match the retailer feed or your pricing system. The stock status should match the warehouse. Name the source of truth for each field explicitly, or you will end up comparing two copies of the same error.

Cross-field

Do fields agree with each other? A product marked as a three-pack should not carry a single-unit barcode. A sale price should be lower than the regular price. A garment in the footwear category is probably miscategorised.

Business rules

Does the record meet your own standards? A minimum of four images per product. No launch before the agreed date. No discount above a set threshold without approval. These rules carry the most value, because nobody outside your organisation will enforce them for you.

Write each rule down with a short name, the fields it touches, the condition for failure and the reason message a reviewer will see. That last part matters more than it seems.

Where checks sit, and what happens when they fail

Place the gate at the release point

Checks belong at the point where a record is about to move from work in progress to live. That is usually just before a feed is sent, a listing is published or a batch is released to a retailer.

Earlier checks are useful too. Validating data as it arrives from a supplier stops bad inputs spreading. But the final gate is the one that protects the customer, so build it first. Run it on every record in the release set, not a sample.

Quarantine failures, do not block everything

A common first design halts the whole release when any record fails. It feels safe. In practice, one missing image on one product delays hundreds of correct records, and the team learns to bypass the check.

The better pattern is quarantine. Records that pass go ahead. Records that fail are held back, and the system records exactly why: missing size attribute, US market or 2 of 4 required images. The person fixing it knows what to do without investigating. Once corrected, the record runs through the checks again and is released with the next batch.

Quarantine turns quality control from a bottleneck into a queue. It also creates a log of every failure, which you will need to measure the system.

Reporting and measurement

Report daily

A check that runs silently is easy to forget. Send a short report every day, automatically, to the people who own the data. Keep it plain: how many records were checked, how many passed, how many were quarantined, and the reasons grouped by rule and market.

The report should answer two questions at a glance. Is today's release safe to go? What needs fixing first? If it takes longer than a minute to read, it is too long.

Measure the system

Three measures tell you whether your checks are working.

Errors intercepted. The number of records quarantined and later corrected before release. This is the clearest evidence of value.
Rule coverage. The proportion of known error types that have a rule. Review past incidents and ask whether a rule would have caught each one. If not, write it.
False positives. Records quarantined that were actually correct. A rule that fires wrongly erodes trust quickly. Track these and tighten the rule rather than tolerating them.

Review these monthly. Retire rules that never fire and add rules for errors that slipped through.

Start small, then scale

You do not need a data platform to begin. A shared spreadsheet with one row per record, a column per rule and a status column is enough to prove the approach. Google Apps Script or a similar tool can run the checks on a schedule, write the reasons into the sheet and email the daily summary.

This is not a toy. Our founder designed and delivered a launch quality system built in Google Apps Script while employed at a global digital-commerce analytics business. It covered more than 30 markets and over 100 retailer lines, validated more than 4,000 records and intercepted more than 180 errors before release, with daily progress reporting fully automated.

Move to heavier tooling when the volume, the number of sources or the speed required outgrows the spreadsheet. By then you will know exactly which rules matter.

A step-by-step rollout plan

List the last ten to twenty errors that reached a live listing. Note the field, the market and how each was found.
Identify the release point: the moment a record goes live. This is where your first checks will run.
Write five to ten rules that would have caught those errors. Give each a name, a condition and a reason message.
Name the source of truth for every field a consistency rule compares.
Build the checks in a spreadsheet or script and run them against last month's data. Confirm they catch the known errors and count the false positives.
Add quarantine. Passing records continue, failing records are held with the reason recorded.
Switch on the daily report and agree who acts on it.
Run alongside manual review for two to four weeks. Compare what each catches.
Reduce manual review to the judgement calls the rules cannot make.
Review the measures monthly and extend rule coverage.

Common mistakes

Writing vague rules. "Images should be good quality" cannot be tested. "At least four images, each at least 1,000 pixels wide" can.
Checking a sample. Errors hide in the records you skipped. Check every record in the release set.
Blocking the whole release. Quarantine the failures and let correct records through.
Unhelpful reason messages. "Validation failed" sends someone hunting. State the rule, the field and the value.
Ignoring false positives. If people learn to override the checks, the system has stopped working.
Never revisiting the rules. Your catalogue, retailers and standards change. The rules must change with them.

If you want to see how this would apply to your own data, we can build a short demo around your process. Details of how builds are scoped and priced are on the services page.

Key takeaways

Manual review fails at volume because of structure, not effort. Automate the repeatable checks first.
Every rule should be testable as pass or fail and carry a clear reason message.
Cover completeness, format, consistency against a named source of truth, cross-field logic and your own business rules.
Quarantine failing records with the reason recorded, rather than blocking the whole release.
Report daily and measure errors intercepted, rule coverage and false positives.
Start with a spreadsheet and a script. Scale the tooling once the rules have proved themselves.

How to catch data errors before they go live