{"id":"data-quality-frameworks","name":"data-quality-frameworks","summary":"Great Expectations、DBTテスト、データ契約を活用してデータ品質検証を実施しましょう。","body":"# Data Quality Frameworks\n\nProduction patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.\n\n## When to Use This Skill\n\n- Implementing data quality checks in pipelines\n- Setting up Great Expectations validation\n- Building comprehensive dbt test suites\n- Establishing data contracts between teams\n- Monitoring data quality metrics\n- Automating data validation in CI/CD\n\n## Core Concepts\n\n### 1. Data Quality Dimensions\n\n| Dimension        | Description              | Example Check                                      |\n| ---------------- | ------------------------ | -------------------------------------------------- |\n| **Completeness** | No missing values        | `expect_column_values_to_not_be_null`              |\n| **Uniqueness**   | No duplicates            | `expect_column_values_to_be_unique`                |\n| **Validity**     | Values in expected range | `expect_column_values_to_be_in_set`                |\n| **Accuracy**     | Data matches reality     | Cross-reference validation                         |\n| **Consistency**  | No contradictions        | `expect_column_pair_values_A_to_be_greater_than_B` |\n| **Timeliness**   | Data is recent           | `expect_column_max_to_be_between`                  |\n\n### 2. Testing Pyramid for Data\n\n```\n          /\\\n         /  \\     Integration Tests (cross-table)\n        /────\\\n       /      \\   Unit Tests (single column)\n      /────────\\\n     /          \\ Schema Tests (structure)\n    /────────────\\\n```\n\n## Quick Start\n\n### Great Expectations Setup\n\n```bash\n# Install\npip install great_expectations\n\n# Initialize project\ngreat_expectations init\n\n# Create datasource\ngreat_expectations datasource new\n```\n\n```python\n# great_expectations/checkpoints/daily_validation.yml\nimport great_expectations as gx\n\n# Create context\ncontext = gx.get_context()\n\n# Create expectation suite\nsuite = context.add_expectation_suite(\"orders_suite\")\n\n# Add expectations\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToNotBeNull(column=\"order_id\")\n)\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeUnique(column=\"order_id\")\n)\n\n# Validate\nresults = context.run_checkpoint(checkpoint_name=\"daily_orders\")\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.\n\n## Summary: {total_passed}/{total_tables} tables passed\")\n        report.append(\"\")\n\n        for table, result in results.items():\n            status = \"✅\" if result.passed else \"❌\"\n            report.append(f\"### {status} {table}\")\n            report.append(f\"- Expectations: {result.total_expectations}\")\n            report.append(f\"- Failed: {result.failed_expectations}\")\n\n            if not result.passed:\n                report.append(\"- Failed checks:\")\n                for detail in result.details:\n                    if not detail[\"success\"]:\n                        report.append(f\"  - {detail['expectation']}: {detail['observed_value']}\")\n            report.append(\"\")\n\n        return \"\\n\".join(report)\n\n# Usage\ncontext = gx.get_context()\npipeline = DataQualityPipeline(context)\n\ntables_to_validate = {\n    \"orders\": \"orders_suite\",\n    \"customers\": \"customers_suite\",\n    \"products\": \"products_suite\",\n}\n\nresults = pipeline.run_all(tables_to_validate)\nreport = pipeline.generate_report(results)\n\n# Fail pipeline if any table failed\nif not all(r.passed for r in results.values()):\n    print(report)\n    raise ValueError(\"Data quality checks failed!\")\n```\n\n## Best Practices\n\n### Do's\n\n- **Test early** - Validate source data before transformations\n- **Test incrementally** - Add tests as you find issues\n- **Document expectations** - Clear descriptions for each test\n- **Alert on failures** - Integrate with monitoring\n- **Version contracts** - Track schema changes\n\n### Don'ts\n\n- **Don't test everything** - Focus on critical columns\n- **Don't ignore warnings** - They often precede failures\n- **Don't skip freshness** - Stale data is bad data\n- **Don't hardcode thresholds** - Use dynamic baselines\n- **Don't test in isolation** - Test relationships too","author":"@wshobson","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/wshobson/agents/tree/main/plugins/data-engineering/skills/data-quality-frameworks","license":"MIT","category":"testing","lang":"en","tokens":868,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/details.md","size":11648,"sha256":"224757dd2c96e2c13fddea88cfe74297f35bf7aae00ec1622bfec0585210f812"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":[]}}