Bad data is not expensive where it enters. It is expensive where it is believed. A malformed row costs nothing at ingestion, a little at transformation, and a great deal once it has fanned out into dashboards, models, and a figure quoted in a board meeting. By the time anyone notices, the row has descendants.

That is why validation belongs upstream. In 0plus, quality gates stand at the front of every pipeline. They fail loudly and early, they divert bad records into quarantine instead of deleting them, and every decision they make is written to the audit log. This post describes the pattern and the rule design behind it.

Validation belongs upstream

Most teams check quality downstream, because downstream is where the pain shows up. A dashboard looks wrong, someone traces it back, and a fix is patched into the reporting layer. The defect stays in the source, ready to leak into the next report. Downstream checks are lagging indicators. They tell you that bad data has already spread, and they say nothing about where else it went.

A gate at the point of entry inverts this. When a batch fails at ingestion, the blast radius is one batch. Nothing has joined it, nothing has aggregated it, and no one has quoted it. The fix is cheap because the defect is still in one place and still looks like whatever the source system produced. Every step a bad record travels multiplies the cost of removing it.

Failing loudly matters as much as failing early. A gate that logs a warning and waves the batch through is decoration. In our pattern, a hard failure halts publication, isolates the batch, and notifies the pipeline owner with the rule that fired and the rows that fired it. Silent coercion is worse than failure. A loader that quietly turns bad dates into nulls has not cleaned the data. It has hidden the evidence.

The quarantine zone

Records that fail a gate do not disappear, and they do not block the whole run. They are diverted into a quarantine table that keeps the original raw values, the rule that rejected them, a reason code, and the batch they arrived in. The clean rows continue on their way. The failed rows wait, visible, in a queue that has an owner.

Quarantine gives quality debt an address. Instead of a vague feeling that the customer table is unreliable, there is a countable backlog: rows in quarantine, the rules that put them there, and the sources that keep producing them. Some fixes belong in the source system. Some are amendments to the rule itself, because sometimes the rule is wrong, not the data. Either way, nothing enters the published layer of the lakehouse without passing the gate, so certification keeps its meaning.

Designing the rules

We group rules into four families, ordered roughly by how cheaply they fail.

  • Schema rules: types, required fields, and column shape. The cheapest checks and the first line of defense, because one schema break upstream means a hundred broken transforms downstream.
  • Range and distribution rules: values inside plausible bounds, volumes inside expected windows. A file with negative quantities, or three times the usual row count, deserves a human eye before it deserves a join.
  • Referential rules: every foreign key resolves, every code exists in its reference table. These catch the quiet failures where a join silently drops rows and a total shrinks without a single error.
  • Arabic text normalization rules: unify alef and hamza variants, normalize taa marbuta, strip tatweel, reconcile Arabic-Indic and Latin digits, and standardize date forms before any matching runs.

The last family is where Arabic-first stops being a slogan. To a join, two spellings of the same customer name are two customers. A gate that normalizes Arabic text at entry, and quarantines what it cannot confidently normalize, is the difference between removing duplicates and manufacturing them. Doing this once at the gate beats doing it at query time in every consumer forever. As an example, a simple rule can be as plain as this:

-- gate: staging_orders, rule qg-014, version 3
SELECT order_id
FROM   staging_orders
WHERE  amount <= 0
   OR  order_date > CURRENT_DATE
   OR  customer_id NOT IN (SELECT id FROM ref_customers);
-- action: quarantine matching rows
-- halt publish when failures exceed 0.5% of the batch

Gates feed the audit log

Every gate decision is an event written at execution time: the rule that ran and its version, the batch it evaluated, how many rows passed, failed, or were quarantined, and who approved the last change to the rule. Rules are versioned policy. When a threshold moves, the audit log shows who moved it, when, and what it was before.

This is where gates meet lineage. Every answer in 0plus carries its lineage, and that lineage carries the gate results of every table it passed through. When an auditor asks how you know a figure is right, the answer is not a story about diligence. It is a query: these rules, these versions, this pass rate, this batch, inside this perimeter, with zero egress to public AI.

A quality rule that fails silently is not a rule. It is a disclaimer that nobody read.

None of this requires a record to leave your environment. The gates, the quarantine zone, and the audit log all run inside the perimeter, so the boundary that keeps the data private is the same boundary that keeps it clean. Validation upstream, failure out loud, quarantine with an owner, and an audit log written as it happens. Bad data will still arrive. It just stops spreading.