Most data quality systems don’t die of bad checks. They die of a missing field in the rule model: the severity. A missing country code in three out of 80,000 rows blocks the nightly load, someone switches the check off “temporarily”, and from that moment everything runs unchecked. Steering data quality with severity levels instead of binary … Read more
The rule “country_code is mandatory” lives in your database twice: once as NOT NULL in the target table’s schema, and once as a hand-typed row in the check configuration. On the next ALTER TABLE, only one of the two places changes, and the check silently goes wrong. With derived data quality rules you no longer type that repetition: the metadata already … Read more
A lot gets written about data quality, and very little gets measured. The German-language practitioner’s standard reference alone lists sixty possible quality criteria — from timeliness to reliability —, and even the lean models still arrive at six to fifteen dimensions. Yet the core of the matter is surprisingly tangible: a data error caught during … Read more
A foreign key pointing into the void is quick to describe — a country_code with no matching entry in the reference data — and surprisingly easy to check incorrectly. The most intuitive phrasing is, of all things, the most dangerous: NOT IN (SELECT …) reads like plain English but silently collapses the moment the reference column holds a single NULL. The … Read more
GROUP BY … HAVING count(*) > 1 may be the most-typed SQL snippet in the world — and yet it trips up in two places almost nobody has on their radar: composite keys and NULL. When it comes to missing values in a unique column, SQL Server and Postgres disagree — one allows exactly one NULL, the other any … Read more
A range check that runs green is no proof of clean data. Anyone who writes WHERE age < 0 OR age > 120 to find implausible ages silently misses every row where age has no value at all — because in SQL, a comparison with NULL is neither true nor false, but unknown. That very missing required value later breaks the load into the strictly … Read more
Bad data gives no warning. An age of 200 years, a duplicate customer number, a country code that doesn’t exist — in the source system nobody notices. Only when the ETL run tries to push the rows into the strictly modelled target layer does the load break: on a CHECK, on a UNIQUE index, on a foreign key. Checking … Read more
A single value that cannot be converted — a date in the wrong format, a number with the wrong decimal separator — and the entire ETL run aborts. Data quality in an ETL process means catching such errors proactively: identifying, logging and isolating them before they reach the target system. This article is the entry point to … Read more
