Lead QA IT Engineer

Key Responsibilities

- Lead QA strategy and define testing approaches for agentic AI systems and ML applications

- Design and implement comprehensive test automation frameworks using Python

- Develop specialized testing methodologies for AI models including accuracy fairness robustness and drift detection

- Build and maintain CI/CD pipelines with integrated automated testing

- Test complex Azure architectures including microservices serverless functions and distributed systems

- Validate data pipelines ETL processes and analytics workflows in Azure Databricks

- Perform advanced performance testing load testing and scalability validation

- Establish quality metrics KPIs and reporting dashboards for AI applications

- Mentor junior QA engineers and promote best practices across the team

- Collaborate with architects and developers on testability and quality requirements

- Implement monitoring and validation strategies for production AI systems

- Lead root cause analysis for critical production issues

- Evaluate and integrate new testing tools and frameworks

- Ensure compliance with security privacy and regulatory requirements

Required Skills & Qualifications

QA Leadership:

- 8 years of experience in software QA with at least 2 years testing AI/ML systems

- Proven track record of establishing QA processes and testing frameworks

- Experience leading testing efforts for production-grade applications

- Strong understanding of QA best practices methodologies and industry standards

Use LLM or AI for testing or Python working experience in testing.

Technical Expertise:

- Expert-level proficiency in Python for test automation and scripting

- Advanced SQL skills for complex data validation and quality checks

- Deep knowledge of Azure services (Functions ML OpenAI Databricks Storage API Management)

- Extensive hands-on experience with Azure Databricks including testing data workflows

- Strong experience building and maintaining CI/CD pipelines (Azure DevOps GitHub Actions)

- Proficiency with test automation frameworks (pytest etc...)

- Experience with API testing and microservices validation

AI/ML Testing Expertise:

- Strong understanding of Machine Learning Natural Language Processing and Deep Learning

- Specialized knowledge of AI/ML testing strategies (model evaluation data quality bias detection adversarial testing)

- Experience testing LLM-based applications including prompt validation and response quality

- Knowledge of MLOps practices and testing ML pipelines

- Understanding of A/B testing and experimentation frameworks

- Familiarity with model monitoring and drift detection

Azure & Databricks:

- Deep knowledge of Azure architecture and deployment patterns

- Experience testing serverless applications and event-driven systems

- Proficiency with Azure monitoring tools (Application Insights Log Analytics)

- Strong understanding of Databricks notebooks jobs and cluster configurations

- Experience with infrastructure testing and validation

Problem-Solving & Quality:

- Exceptional analytical and debugging skills

- Experience with performance testing tools.

- Knowledge of security testing principles and tools

- Strong understanding of observability and monitoring strategies

Preferred Qualifications

- Bachelors or Masters degree in Computer Science Engineering or related field

- Azure certifications (Azure Administrator Azure Solutions Architect)

- Databricks certification (Spark Developer ML Associate)

- Experience with resilience testing

- Experience testing multi-agent systems or autonomous AI applications

- Familiarity with MLflow Azure ML or other MLOps platforms

- Experience with test data management and synthetic data generation