Lesson · Hinglish · ⏱ 6 min padhne mein

🧹 Data quality: kharab data jaldi pakadna

Q: Kharab value pakadne ki sabse sasti jagah kaunsi hai?

Pipeline mein jaldi, automated checks se. Jitna jaldi pakdoge, utne kam systems usse kharab honge.

🏠 Asli zindagi ki misaal

Achha restaurant sabziyaan kitchen tak pahunchne se pehle hi back door pe check karta hai. Wahan pakde gaye sade tamatar ka koi nuksaan nahi; customer ki plate pe pakde gaye to restaurant ki izzat ka nuksaan. Data quality checks data ke liye wahi back-door inspection hain.

💡 Aasaan shabdon mein

Kharab data chupchaap models aur dashboards tod deta hai. Pipeline ke har stage pe data automatically check karo: completeness (unexpected nulls nahi), validity (umar 0 se 120 ke beech, sahi email format), uniqueness (duplicate order IDs nahi), consistency (totals unke line items se match karein), aur freshness (data time pe aaya). Great Expectations ya dbt tests jaise tools in rules ko code mein likhte hain, aur rule tootne pe pipeline fail karte hain ya alert bhejte hain. ML ke liye label quality aur class balance bhi check karo, aur upstream teams ke saath data contract rakho, taaki schema changes achanak pareshaan na karein.

🪜 Step by step

Schema check karo: expected columns aur types aaye.
Volume check karo: row counts haal ke dino ke mukaable normal range mein.
Values check karo: required fields mein NULL nahi, values allowed range mein, unique keys sach mein unique.
Freshness check karo: sabse naya data kaafi recent hai.
Fail hone pe kya hoga tay karo: critical issues pe pipeline roko, chhote pe alert, aur data use karne walon ko batao.

🔑 Zaroori shabd

Validation rule: Check jo data ko pass karna hi hai, jaise amount ≥ 0.
Freshness: Data utna naya hai ya nahi jitna hona chahiye.
Data contract: Data bhejne aur use karne wali teams ke beech schema aur quality ka agreement.

👀 Example dekho

yaml

# dbt tests on the orders model
models:
  - name: fact_orders
    columns:
      - name: order_id
        tests: [unique, not_null]
      - name: amount
        tests:
          - dbt_utils.accepted_range: { min_value: 0 }
      - name: status
        tests:
          - accepted_values: { values: ['placed', 'delivered', 'cancelled'] }

Rules data ke saath rehte hain aur har pipeline run pe chalte hain.

⚠️ Aam galtiyaan jinse bachna hai

Quality sirf aakhir mein check karna. Problems wahin pakdo jahan data aata hai, failne se pehle.
Har data source ka owner nahi, jisse failures teams ke beech ghoomte rehte hain.

✅ Jaldi se check karo

Kharab value pakadne ki sabse sasti jagah kaunsi hai?

CEO ke dashboard mein
Pipeline mein jaldi, automated checks se
Model retrain hone ke baad

Jawaab dekho

B. Pipeline mein jaldi, automated checks se: Jitna jaldi pakdoge, utne kam systems usse kharab honge.

🙋 Seekhne wale yeh poochhte hain

Kaunse tools madad karte hain?

dbt tests, Great Expectations, Soda aur warehouse ke built in constraints. Kuch SQL checks se simple shuru karo.

Data contract kya hai?

Data banane wali team aur use karne wali teams ke beech schema, matlab aur quality ka agreement, taaki changes chupke se cheezein na todein.

📌 Yaad rakho

Completeness, validity, uniqueness, consistency aur freshness automatically check karo.

📖 Is lesson ke shabd (AI dictionary)

Poora course, aapke roadmap ke hisaab se

Progress tracking, quick checks aur step-by-step order ke saath. Yeh lesson in courses ka hissa hai:

Data Engineer (AI focus)

Course mein seekho