QA Lead for AI Project

Bengaluru - India

Monthly Salary: Not Disclosed

Posted on: 4 hours ago

Vacancies: 1 Vacancy

Job Summary

What is expected
Design automate and execute API test suites (functional contract
performance resilience)
Define LLM evaluation strategies (performance testing golden sets guardrails
hallucination/consistency checks relevance & latency KPIs); build automated
eval pipelines tied to releases.
Create PDLM deliverables: verification plans/protocols test reports risk-based
coverage V&V trace matrices to requirements and use cases.
Integrate test infrastructure with Azure DevOps (CI/CD test orchestration
quality gates dashboards); enforce build-time gates for API and LLM quality.
Partner with Product Clinical and R&D to refine requirements and acceptance
criteria; ensure summaries render correctly for different user roles and
workflows.

What is required from candidate
8 years in backend/API test automation (REST JSON auth) performance
testing and CI/CD.
Hands-on with Python/TypeScript Postman/Newman or REST-assured
PyTest/Jest k6/JMeter Docker.
Experience validating ML/LLM systems (eval datasets metrics prompt/response
testing safety/guardrails).
Strong documentation and traceability discipline for regulated product lifecycles
(PDLM/QMS).
Bonus: Azure DevOps pipelines FHIR/HL7 familiarity healthcare data
privacy/security.
Bonus: Frontend testing capabilities are preferred but not part of the PDLM
requirements.

Interested candidates can share their updated resume to rolly..com

What is expected Design automate and execute API test suites (functional contract performance resilience) Define LLM evaluation strategies (performance testing golden sets guardrails hallucination/consistency checks relevance & latency KPIs); build automated eval pipelines tied to releases. ...