DE&A Core Data Quality Management Data Quality Management (Other)

We are seeking a hands on QA Lead to drive quality assurance for a scalable enterprise-wide data platform for an insurance client. The role involves validating batch and real-time data pipelines ensuring data accuracy across Raw Silver and Gold layers and supporting report rationalization and self-service analytics (Power BI).

The QA Lead will define and implement end-to-end data testing strategies covering ingestion (AWS Glue Kinesis) transformation (DBT) and consumption layers while ensuring data quality integrity and performance optimization.

Key Responsibilities

1. QA Strategy & Leadership

• Define and implement end-to-end QA strategy for the enterprise data platform
• Establish test frameworks standards and governance for data validation
• Lead QA planning estimation and execution across multiple data streams

2. Data Validation & Testing

• Validate data across Raw Silver and Gold layers ensuring accuracy completeness and consistency
• Perform source-to-target reconciliation for batch and real-time pipelines
• Design and execute:
  • Data quality checks
  • Transformation validation (DBT models)
  • Aggregation and KPI validation

3. Batch & Real-Time Pipeline Testing

• Test batch ingestion pipelines using AWS Glue
• Validate real-time streaming data pipelines using Amazon Kinesis
• Ensure data latency sequencing and event consistency in streaming pipelines

4. Reporting & Rationalization QA

• Validate datasets powering Power BI self-service reports
• Support report rationalization initiatives by ensuring consistency of KPIs and eliminating redundant data sources
• Perform report/data reconciliation testing across legacy vs new platform

5. Automation & Tools

• Develop and implement automated data testing frameworks
• Leverage SQL Python and testing tools (e.g. Great Expectations DBT tests custom frameworks)
• Enable continuous testing integration within CI/CD pipelines

6. Performance & Optimization Testing

• Validate performance of:
  • Data pipelines
  • Queries in Snowflake
• Identify bottlenecks and work with engineering teams to optimize pipelines and queries
• Ensure scalability for large data volumes and concurrent workloads

7. Data Quality & Governance

• Define and enforce data quality rules thresholds and monitoring
• Implement data anomaly detection and alerting mechanisms
• Ensure compliance with audit reconciliation and governance standards

Required Skills & Experience

Core Technical Skills

• Strong experience in data testing / ETL testing / data QA
• Hands-on expertise with:
  • Snowflake (data validation SQL testing)
  • DBT (testing model validation)
  • AWS Glue (batch pipeline validation)
  • Amazon Kinesis (real-time pipeline testing)
• Advanced proficiency in SQL for data validation and reconciliation
• Programming skills in Python (preferred)

Testing Expertise

• Experience in:
  • Data reconciliation (source vs target)
  • Data quality frameworks and validation techniques
  • Automated data testing tools
• Understanding of medallion architecture (Raw Silver Gold layers)

Analytics & Reporting

• Experience validating Power BI reports and datasets
• Strong understanding of business KPIs and reporting consistency

Domain Expertise (Preferred)

• Experience in Insurance domain (Policy Claims Billing data)
• Familiarity with regulatory reporting audit and reconciliation requirements

Experience

• 812 years in QA / Data Testing / ETL Testing
• 3 years in QA leadership or lead role
• Experience working on enterprise-scale data platforms

QA Lead

QA Lead