Gen AI QA Lead

Company: WillWare Technologies
Role: Gen AI QA Lead
Experience: 6 Years
Location: Bangalore Hyderabad Pune
Work Mode: Hybrid

Position Summary:
As a GenAI Test Lead you will define and operationalize quality for AI systems bridging traditional QA with AI engineering. You will design scalable evaluation frameworks (EvalOps) to measure groundedness factual accuracy coherence and RAG performance ensuring our AI systems are reliable aligned and production-ready.
This role requires treating LLM outputs as measurable signalsbuilding data-driven validation systems that continuously improve model performance and user experience.

Key responsibilities and duties -

• Define evaluation metrics for LLM systems (groundedness hallucination rate task success RAG accuracy)
• Design and implement automated EvalOps pipelines for continuous model validation
• Establish acceptance criteria and quality benchmarks for AI releases
• Build scalable test frameworks for LLM-based systems using tools like DeepEval pytest or custom pipelines
• Automate validation of prompts responses and agent workflows
• Integrate AI testing into CI/CD pipelines
• Analyze LLM outputs as structured data to identify failure patterns
• Validate data quality retrieval accuracy and grounding in RAG systems
• Design synthetic and real-world test datasets
• Collaborate with AI engineers product teams and clients to define quality standards
• Drive alignment across distributed/onshore-offshore teams
• Provide insights on model performance and improvement areas
• Evaluate prompt strategies agent behaviors and user interaction flows
• Ensure outputs align with business context and user expectations
• Rapidly adapt to domain-specific requirements
• Experience with Rational Tool Suite JIRA or similar test management tools.
• Experience in using/setting up continuous integration testing and experience with any DevOps tools
• Develop automation tests as part of the Eval Framework using frameworks like DeepEval pytest etc
• Strong understanding ofAgile principles;experience in SAFe Agile environments.
• Experience testing APIs and data integrations.
• Experience working in anonshore/offshore model.
• Experience intest automationdesign and execution.
• Strong understanding of different software testing methodologies.

Required Qualifications and Experience:

• Minimum 6 years of overall experience in software QA.
• Education: Bachelors or Masters degree in Computer Science Data Science Engineering or a related quantitative field.
• Problem-Solving: Strong analytical and problem-solving skills with the ability to troubleshoot complex distributed AI systems.
• Communication: Excellent communication skills to articulate technical findings and development progress effectively.

Required Skills:

AutomationTestingArtificial Intelligence