QA Engineer
Job Location:
Monthly Salary:
Posted on:
15 hours ago
Posted on:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
QA Engineer - AI Initiatives
Role Overview
EisnerAmper is seeking a talentedQA Engineer AI Initiativesto join our growing this role you will be responsible for ensuring the quality reliability fairness and performance of AI/ML-powered products and systems. Unlike traditional QA this position requires a deep understanding of non-deterministic model behavior data quality and AI-specific failure modes such as hallucinations bias and model drift. You will be at the forefront of AI quality assurance collaborating with data scientists ML engineers and product teams to deliver robust ethical and high-performing AI solutions.
Key Responsibilities
Design and execute comprehensive test strategies specifically for AI/ML models LLM-based applications and data pipelines
Develop automated test frameworks for model validation regression testing and performance benchmarking
Evaluate model outputs for accuracy consistency relevance hallucination and bias across diverse inputs and use cases
Test RAG (Retrieval-Augmented Generation) pipelines chatbots recommendation systems and other AI-driven features
Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds for AI systems
Build and maintain evaluation datasets ground truth sets and adversarial test cases for comprehensive model validation
Monitor models in production for drift degradation and anomalous behavior; implement monitoring solutions as needed
Validate data quality data pipelines and feature stores that feed AI systems to ensure data integrity
Document defects edge cases and failure patterns specific to AI behavior with actionable insights
Ensure AI systems meet ethical fairness and compliance standards through bias audits and explainability checks
Required Skills & Qualifications
Bachelors or Masters degree in Computer Science Engineering or a related field
3 - 6 years of professional QA experience with at least 1 - 2 years in AI/ML quality assurance
Strong proficiency inPythonfor test automation and data analysis
Familiarity with LLM evaluation frameworks (e.g. RAGAS DeepEval Promptfoo LangSmith)
Hands-on experience with testing tools such as Pytest Selenium Postman or similar platforms
Solid understanding of the ML lifecycle â training validation deployment and monitoring phases
Knowledge of data quality tools and pipeline testing (e.g. Great Expectations dbt tests)
Strong analytical and inquisitive mindset with the ability to challenge model outputs critically
Excellent documentation and communication skills with the ability to articulate complex technical concepts
Collaborative approach and ability to work effectively with data science engineering and product teams
Nice to Have
Experience with prompt engineering and red-teaming LLMs
Familiarity with MLOps platforms such as MLflow SageMaker or Vertex AI
Knowledge of vector databases and embedding quality evaluation
Understanding of AI safety responsible AI principles and fairness frameworks
Experience with A/B testing and shadow deployment strategies
Knowledge of CI/CD pipelines and DevOps practices in ML environments
Role Overview
EisnerAmper is seeking a talentedQA Engineer AI Initiativesto join our growing this role you will be responsible for ensuring the quality reliability fairness and performance of AI/ML-powered products and systems. Unlike traditional QA this position requires a deep understanding of non-deterministic model behavior data quality and AI-specific failure modes such as hallucinations bias and model drift. You will be at the forefront of AI quality assurance collaborating with data scientists ML engineers and product teams to deliver robust ethical and high-performing AI solutions.
Key Responsibilities
Design and execute comprehensive test strategies specifically for AI/ML models LLM-based applications and data pipelines
Develop automated test frameworks for model validation regression testing and performance benchmarking
Evaluate model outputs for accuracy consistency relevance hallucination and bias across diverse inputs and use cases
Test RAG (Retrieval-Augmented Generation) pipelines chatbots recommendation systems and other AI-driven features
Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds for AI systems
Build and maintain evaluation datasets ground truth sets and adversarial test cases for comprehensive model validation
Monitor models in production for drift degradation and anomalous behavior; implement monitoring solutions as needed
Validate data quality data pipelines and feature stores that feed AI systems to ensure data integrity
Document defects edge cases and failure patterns specific to AI behavior with actionable insights
Ensure AI systems meet ethical fairness and compliance standards through bias audits and explainability checks
Required Skills & Qualifications
Bachelors or Masters degree in Computer Science Engineering or a related field
3 - 6 years of professional QA experience with at least 1 - 2 years in AI/ML quality assurance
Strong proficiency inPythonfor test automation and data analysis
Familiarity with LLM evaluation frameworks (e.g. RAGAS DeepEval Promptfoo LangSmith)
Hands-on experience with testing tools such as Pytest Selenium Postman or similar platforms
Solid understanding of the ML lifecycle â training validation deployment and monitoring phases
Knowledge of data quality tools and pipeline testing (e.g. Great Expectations dbt tests)
Strong analytical and inquisitive mindset with the ability to challenge model outputs critically
Excellent documentation and communication skills with the ability to articulate complex technical concepts
Collaborative approach and ability to work effectively with data science engineering and product teams
Nice to Have
Experience with prompt engineering and red-teaming LLMs
Familiarity with MLOps platforms such as MLflow SageMaker or Vertex AI
Knowledge of vector databases and embedding quality evaluation
Understanding of AI safety responsible AI principles and fairness frameworks
Experience with A/B testing and shadow deployment strategies
Knowledge of CI/CD pipelines and DevOps practices in ML environments
Required Skills:
MLQA AutomationAIPython
