allowed-tools: Task, Read, Write, Bash, Grep, Glob

argument-hint: [dataset] [action]

description: Perform data quality validation, checks, and monitoring for specified dataset

Data Quality Command

Execute data quality operations on dataset $1 with action $2 using the quality-assurance subagent.

Context

• Dataset location: @data_storage/$1
• Quality action: $2 (check, clean, validate, monitor, profile)
• Current working directory: !pwd
• Output directory: ./quality_reports/
• Quality rules and validation thresholds
• Available quality metrics and KPIs

Your Task

Use the quality-assurance subagent to perform comprehensive data quality operations:

1. Quality Assessment

• Analyze data completeness and accuracy
• Check data consistency and validity
• Assess data uniqueness and timeliness
• Evaluate overall data integrity

2. Issue Identification

• Detect missing values and data gaps
• Identify outliers and anomalies
• Find duplicate records and inconsistencies
• Discover format violations and data type issues

3. Quality Improvement

• Implement data cleaning procedures
• Apply data validation rules
• Execute data transformation operations
• Perform data standardization

4. Monitoring and Reporting

• Generate quality metrics and KPIs
• Create quality assessment reports
• Set up ongoing quality monitoring
• Provide quality improvement recommendations

Quality Actions

Check

Perform basic data quality assessment:

• Completeness analysis
• Basic accuracy validation
• Simple consistency checks
• Summary quality metrics

Clean

Execute data cleaning operations:

• Remove duplicate records
• Handle missing values
• Correct format violations
• Standardize data formats

Validate

Comprehensive data validation:

• Statistical validation
• Business rule validation
• Cross-field validation
• Referential integrity checks

Monitor

Set up quality monitoring:

• Continuous quality tracking
• Alert threshold configuration
• Quality trend analysis
• Performance metrics monitoring

Profile

Generate comprehensive data profile:

• Detailed data statistics
• Distribution analysis
• Relationship analysis
• Data lineage documentation

Quality Dimensions

Completeness

• Missing Value Analysis: Identify and quantify missing data
• Required Field Validation: Check presence of mandatory fields
• Record Completeness: Assess completeness of individual records
• Data Coverage: Evaluate coverage of expected data range

Accuracy

• Statistical Validation: Verify statistical properties
• Business Rule Validation: Check against business constraints
• Range Validation: Ensure values within expected ranges
• Format Validation: Verify correct data formats

Consistency

• Cross-Field Validation: Check logical consistency between fields
• Temporal Consistency: Validate time-based consistency
• Referential Integrity: Check relationship consistency
• Format Consistency: Ensure consistent formatting

Timeliness

• Data Currency: Assess how current the data is
• Update Frequency: Evaluate data refresh rates
• Latency Analysis: Measure data processing delays
• Freshness Metrics: Track data age and relevance

Uniqueness

• Duplicate Detection: Identify and eliminate duplicate records
• Primary Key Validation: Verify unique identifiers
• Record Uniqueness: Assess overall uniqueness
• Relationship Uniqueness: Check unique relationships

Validity

• Data Type Validation: Verify correct data types
• Domain Validation: Check against allowed value domains
• Pattern Validation: Validate against expected patterns
• Constraint Validation: Check database and business constraints

Expected Output

Quality Reports

• quality_reports/$1_quality_check.json - Quality assessment results
• quality_reports/$1_data_profile.json - Comprehensive data profile
• quality_reports/$1_validation_report.md - Detailed validation report
• quality_reports/$1_monitoring_config.json - Monitoring configuration

Quality Metrics

• Overall Quality Score: Composite quality metric (0-100)
• Dimension Scores: Individual quality dimension scores
• Issue Counts: Number and severity of quality issues
• Improvement Metrics: Quality improvement tracking

Data Outputs

• Cleaned Data: Quality-improved dataset versions
• Validation Logs: Detailed validation results
• Error Reports: Specific error descriptions and locations
• Recommendations: Actionable improvement suggestions

Working Process

1. Data Loading and Profiling

import pandas as pd
import numpy as np
from scipy import stats

def load_and_profile_data(dataset_path):
    """Load dataset and create initial profile"""
    data = pd.read_csv(dataset_path)

    profile = {
        'basic_info': {
            'shape': data.shape,
            'columns': list(data.columns),
            'data_types': data.dty