Data QA Engineer (Databricks)

Data QA Engineer Databricks Development & Migration Projects

Supporting Databricks Initiatives on AWS and Azure Cloud Platforms

We are seeking an experienced Data QA Engineer to join our team and support Databricks development and migration projects. The ideal candidate will have a minimum of 5 years of hands-on experience working with Databricks on cloud platforms such as AWS and this role you will collaborate closely with data engineering analytics and migration teams to ensure the highest quality standards for our data solutions and migration processes.

• Design develop and execute comprehensive test strategies for Databricks-based data pipelines ETL processes and migration solutions on AWS and Azure.
• Develop and maintain automated test scripts to validate data accuracy transformation logic and system performance within Databricks environments.
• Work closely with developers data engineers and business analysts to define test requirements acceptance criteria and data validation protocols.
• Analyze data migration results identify discrepancies and drive resolution of data quality issues.
• Perform root cause analysis of defects and provide detailed reports to stakeholders.
• Contribute to the continuous improvement of QA processes test automation frameworks and best practices for Databricks on cloud platforms.
• Participate in code reviews design discussions and sprint planning activities to advocate for quality standards.

• 5 years of experience working with Databricks on cloud platforms (AWS and/or Azure).
• Strong expertise in data quality assurance testing methodologies and test automation for big data and cloud-based environments.
• Hands-on experience in writing and executing SQL queries for data validation and analysis.
• Proficiency with PySpark Python and Databricks notebooks for test automation and data validation.
• Experience in validating ETL pipelines data transformations and large-scale data migrations.
• Solid understanding of cloud storage solutions (e.g. AWS S3 Azure Data Lake) and data integration patterns.
• Proven ability to troubleshoot and resolve data quality issues in complex distributed environments.
• Excellent communication and collaboration skills.

• Experience with CI/CD tools for data pipeline deployment and automated testing (e.g. Jenkins Azure DevOps).
• Familiarity with other data platforms and tools such as Snowflake Redshift or Synapse Analytics.
• Knowledge of data governance data lineage and metadata management.
• Experience with performance testing and optimization of big data workloads.
• Exposure to scripting languages such as Scala or Shell scripting.
• Relevant certifications (e.g. Databricks Certified Data Engineer AWS Certified Solutions Architect Azure Data Engineer Associate).

• Bachelors or Masters degree in Computer Science Information Systems Engineering or a related field.