Senior AI QA Analyst

• Implement AI quality strategies for LLM RAG and agentic workflowsfrom test design through post-launch monitoring.

• Design and execute test plans combining both traditional QA and AI-focused evaluation for complex non-deterministic systems.

• Validate data quality through schema checks drift/shift detection and ground-truth audits; drive remediation with DS and Engineering.

• Build adversarial and synthetic test sets to stress-test reasoning chains tool-usage hallucinations jailbreaks and prompt-injection risks.

• Develop and maintain evaluation pipelines to measure grounding robustness calibration cost latency task-completion and system reliability.

• Instrument production systems for continuous monitoring including risk taxonomies alerting criteria escalation pathways and quality guardrails.

• Establish and refine guardrails for AI features (content moderation input/output validation tool-call limits loop-prevention checks and more).

• Create and maintain comprehensive test cases user scenarios and documentation for both AI behaviors and traditional software components.

• Collaborate with Data Scientists and AI Engineers to interpret model performance evaluation metrics and overall system behavior.

• Report quality health through clear dashboards documentation and incident reviews while influencing roadmaps through evidence-based recommendations.

• Contribute to continuous improvement by codifying templates pipelines and evaluation frameworks to scale QA practices across teams.

Qualifications :

• 45 years of experience in QA including manual and automated testing in software environments.

• At least 1 year of experience working with or supporting Data Science ML AI or analytics workflows.

• Basic Python knowledge with familiarity using common DS libraries (NumPy Pandas) and Pytest for unit testing.

• Practical understanding of AI/LLM evaluation including prompt/test design evaluation rubrics and interpretation of model behavior.

• Experience assessing data quality detecting drift or inconsistencies and performing ground-truth verification.

• Ability to design synthetic/adversarial test sets for AI-driven applications.

• Basic understanding of cloud environments and modern data/AI pipelines.

• Familiarity with QA automation concepts and integration/regression/performance testing.

• Strong ability to translate ambiguous model outputs into testable hypotheses measurable metrics and clear action steps.

• Experience working with Jira and Confluence for test planning documentation and workflow management.

• Basic SQL and Excel skills to support data validation and reporting.

• Excellent communication analytical thinking and cross-functional collaboration skills.

What about languages 

You will need excellent written and verbal English for clear and effective communication with the team. 

How much experience must I have 

• Were looking for someone with 4 years of experience in similar roles. 

Additional Information :

Our perks and benefits:  

Every day lunches! (headquarters): 

• Vegetarian vegan gluten and sugar free options.  

• Gourmet meals every Friday with our on-site chef! 

Flexible working options to help you strike the right balance.  

All the equipment you need to harness your talent (Macbook and accessories).  

Snacks and beverages available everyday (headquarters).  

After office events football tennis and game nights (headquarters).  

• Everyone is welcome to join our football league every Wednesdays and Fridays.  

• Challenge your teammates to a pool game and win the offices trophy!  

• Tennis courts available for friendly matches.  

• You are not a sports person Dont worry we also have chess championships game and music nights for you to join!  

Learning opportunities:  

• AWS Certifications (we are AWS Partners).  

• Study plans courses and other certifications.  

• English Lessons.  

• Learn from your teammates on our Tech Tuesdays!  

Mentoring and Development opportunities to shape your career path.  

Anniversary and birthday gifts. 

Great location and even greater teammates! 

So what are the next steps 

Our team is eager to learn about you! Send us your resume or LinkedIn profile below and well explore working together! 

Remote Work :

No

Employment Type :

Full-time