Job Description Summary

Job Description

• Design develop and execute comprehensive benchmark frameworks to evaluate AI use cases prompt performance robustness across diverse array of use cases.
• Develop automated testing suites to validate AI functionalities such as prediction accuracy response consistency case handling bias detection and model degradation over time.
• Collaborate closely with Prompt Engineers AI Agent Engineers and product owners to define quality standards acceptance criteria and key performance indicators (KPIs) for AI deployments.
• Conduct quantitative and qualitative evaluations of AI outputs ensuring alignment with business objectives regulatory standards and ethical AI principles.
• Monitor AI use cases performance in production systematically identify issues and recommend improvements to maintain high standards of AI quality and safety.
• Document testing methodologies framework designs quality metrics and ensure thorough reporting to stakeholders.
• Support continuous improvement initiatives by incorporating feedback loops and new benchmarking techniques as AI technologies evolve.
• Provide training and mentorship on AI testing best practices within GE Vernovas Prompt engineering and AI Agent Engineering team.
• Stay current with emerging AI technologies contribute to platform and tooling improvements and share knowledge within the team.

Required Qualifications and Skills:

• Masters degree in Computer Science AI Engineering Data Science or related field.
• 3 years of experience in software quality assurance with at least 2 years focused on AI/ML systems.
• Proven track record designing and implementing AI benchmark frameworks or quality assurance strategies.
• Strong programming skills in Python and experience with automated testing frameworks (e.g. pytest Selenium).
• Familiarity with RAG systems vector databases and GenAI architectures.
• Deep understanding of machine learning models LLMs and AI validation methodologies.
• Experience with AI fairness bias detection and responsible AI practices.
• Knowledge of cloud environments (AWS Azure or GCP) containerization and deployment pipelines.
• Ability to analyze complex datasets and performance metrics quantitatively and qualitatively.
• Strong communication skills with ability to document and present technical information clearly.
• Hands-on experience with LLMs prompt engineering and natural language processing (NLP).
• Knowledge of agent orchestration platforms and multi-agent systems (e.g. AutogenAI LangGraph MCP protocol).

Additional Information