Lesson · ⏱ 6 min read

🧹 Data quality: catching bad data early

Q: Where is the cheapest place to catch a bad value?

Early in the pipeline, with automated checks. The earlier you catch it, the fewer systems it pollutes.

🏠 Real-life example

A good restaurant checks vegetables at the back door, before they reach the kitchen. Rotten tomatoes caught there cost nothing; caught on a customer's plate, they cost the restaurant's reputation. Data quality checks are that back-door inspection for data.

💡 In simple words

Bad data quietly breaks models and dashboards. Check data automatically at each pipeline stage: completeness (no unexpected nulls), validity (ages between 0 and 120, valid email formats), uniqueness (no duplicate order IDs), consistency (totals match their line items), and freshness (data arrived on time). Tools like Great Expectations or dbt tests codify these rules and fail the pipeline or alert when they break. For ML, also check label quality and class balance, and keep a data contract with upstream teams so schema changes don't surprise you.

🪜 Step by step

Check the schema: expected columns and types arrived.
Check volume: row counts within a normal range compared with recent days.
Check values: no NULLs in required fields, values within allowed ranges, unique keys really unique.
Check freshness: the newest data is recent enough.
Decide what happens on failure: stop the pipeline for critical issues, alert for minor ones, and tell data consumers.

🔑 Key terms

Validation rule: A check data must pass, e.g. amount ≥ 0.
Freshness: Whether data is as recent as expected.
Data contract: An agreement on schema and quality between data producers and consumers.

👀 See it in action

yaml

# dbt tests on the orders model
models:
  - name: fact_orders
    columns:
      - name: order_id
        tests: [unique, not_null]
      - name: amount
        tests:
          - dbt_utils.accepted_range: { min_value: 0 }
      - name: status
        tests:
          - accepted_values: { values: ['placed', 'delivered', 'cancelled'] }

Rules live next to the data and run on every pipeline run.

⚠️ Common mistakes to avoid

Checking quality only at the end. Catch problems where data enters, before they spread.
No owner for each data source, so failures bounce between teams.

✅ Quick check

Where is the cheapest place to catch a bad value?

In the CEO's dashboard
Early in the pipeline, with automated checks
After the model is retrained

Show answer

B. Early in the pipeline, with automated checks: The earlier you catch it, the fewer systems it pollutes.

🙋 Questions learners ask

Which tools help?

dbt tests, Great Expectations, Soda and built in warehouse constraints. Start simple with a few SQL checks.

What is a data contract?

An agreement between the team producing data and the teams using it about schema, meaning and quality, so changes don't break things silently.

📌 Remember

Check completeness, validity, uniqueness, consistency and freshness, automatically.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

Data Engineer (AI focus)

Continue in the course