Data quality (dataset)

--- allowed-tools: Task, Read, Write, Bash, Grep, Glob argument-hint: [dataset] [action] description: Perform data quality validation, checks, and monitoring for specified dataset --- # Data Quality Command Execute data quality operations on dataset $1 with action $2 using the quality-assurance subagent. ## Context - Dataset location: @data_storage/$1 - Quality action: $2 (check, clean, validate, monitor, profile) - Current working directory: !pwd - Output directory: ./quality_reports/ - Quality rules and validation thresholds - Available quality metrics and KPIs ## Your Task Use the quality-assurance subagent to perform comprehensive data quality operations: ### 1. Quality Assessment - Analyze data completeness and accuracy - Check data consistency and validity - Assess data uniqueness and timeliness - Evaluate overall data integrity ### 2. Issue Identification - Detect missing values and data gaps - Identify outliers and anomalies - Find duplicate records and inconsistencies - Discover format violations and data type issues ### 3. Quality Improvement - Implement data cleaning procedures - Apply data validation rules - Execute data transformation operations - Perform data standardization ### 4. Monitoring and Reporting - Generate quality metrics and KPIs - Create quality assessment reports - Set up ongoing quality monitoring - Provide quality improvement recommendations ## Quality Actions ### Check Perform basic data quality assessment: - Completeness analysis - Basic accuracy validation - Simple consistency checks - Summary quality metrics ### Clean Execute data cleaning operations: - Remove duplicate records - Handle missing values - Correct format violations - Standardize data formats ### Validate Comprehensive data validation: - Statistical validation - Business rule validation - Cross-field validation - Referential integrity checks ### Monitor Set up quality monitoring: - Continuous quality tracking - Alert threshold configuration - Quality trend analysis - Performance metrics monitoring ### Profile Generate comprehensive data profile: - Detailed data statistics - Distribution analysis - Relationship analysis - Data lineage documentation ## Quality Dimensions ### Completeness - Missing Value Analysis: Identify and quantify missing data - Required Field Validation: Check presence of mandatory fields - Record Completeness: Assess completeness of individual records - Data Coverage: Evaluate coverage of expected data range ### Accuracy - Statistical Validation: Verify statistical properties - Business Rule Validation: Check against business constraints - Range Validation: Ensure values within expected ranges - Format Validation: Verify correct data formats ### Consistency - Cross-Field Validation: Check logical consistency between fields - Temporal Consistency: Validate time-based consistency - Referential Integrity: Check relationship consistency - Format Consistency: Ensure consistent formatting ### Timeliness - Data Currency: Assess how current the data is - Update Frequency: Evaluate data refresh rates - Latency Analysis: Measure data processing delays - Freshness Metrics: Track data age and relevance ### Uniqueness - Duplicate Detection: Identify and eliminate duplicate records - Primary Key Validation: Verify unique identifiers - Record Uniqueness: Assess overall uniqueness - Relationship Uniqueness: Check unique relationships ### Validity - Data Type Validation: Verify correct data types - Domain Validation: Check against allowed value domains - Pattern Validation: Validate against expected patterns - Constraint Validation: Check database and business constraints ## Expected Output ### Quality Reports - quality_reports/$1_quality_check.json - Quality assessment results - quality_reports/$1_data_profile.json - Comprehensive data profile - quality_reports/$1_validation_report.md - Detailed validation report - quality_reports/$1_monitoring_config.json - Monitoring configuration ### Quality Metrics - Overall Quality Score: Composite quality metric (0-100) - Dimension Scores: Individual quality dimension scores - Issue Counts: Number and severity of quality issues - Improvement Metrics: Quality improvement tracking ### Data Outputs - Cleaned Data: Quality-improved dataset versions - Validation Logs: Detailed validation results - Error Reports: Specific error descriptions and locations - Recommendations: Actionable improvement suggestions ## Working Process ### 1. Data Loading and Profiling ```python import pandas as pd import numpy as np from scipy import stats def loadandprofiledata(datasetpath): """Load dataset and create initial profile""" data = pd.readcsv(datasetpath) profile = { 'basicinfo': { 'shape': data.shape, 'columns': list(data.columns), 'datatypes': data.dty