AI SDET – AI Testing Test Automation

Job Description: AI SDET AI Testing / Test Automation
Location: Sunnyvale CA
Job Title: AI Software Development Engineer in Test (AI SDET)
Rate: $55 -$65/hr
Job Summary
We are seeking an experienced AI Software Development Engineer in Test (AI SDET) to design develop and implement comprehensive testing strategies for AI-powered applications machine learning models and Generative AI systems. The ideal candidate will have strong experience in software test automation API testing Python programming and AI/ML testing methodologies.
The candidate will be responsible for validating the functionality performance reliability accuracy and security of AI-driven solutions including Large Language Models (LLMs) Retrieval-Augmented Generation (RAG) pipelines AI agents and intelligent automation workflows. This role requires close collaboration with software developers data scientists machine learning engineers and product teams to ensure high-quality AI products.
Key Responsibilities

• Design develop and maintain scalable automated test frameworks for AI/ML applications using Python and modern automation tools.
• Create and execute test plans test cases and test strategies for AI-powered applications APIs and backend services.
• Validate Generative AI and LLM-based applications for accuracy relevance consistency hallucinations bias toxicity and response quality.
• Develop automated evaluation frameworks to measure AI model performance precision recall latency and reliability.
• Test RAG-based applications including document ingestion chunking embedding generation vector database retrieval and response generation.
• Perform API testing integration testing end-to-end testing regression testing and system testing.
• Build automation scripts using Python Pytest Selenium Playwright or equivalent testing frameworks.
• Validate AI model outputs using predefined datasets evaluation metrics assertions and quality thresholds.
• Test AI agents and tool-calling workflows for correct decision-making tool selection execution and error handling.
• Conduct performance load stress and scalability testing for AI services and inference APIs.
• Integrate automated tests into CI/CD pipelines using Jenkins GitHub Actions GitLab CI or similar tools.
• Test AI application security including prompt injection jailbreak attempts sensitive data exposure and unsafe model outputs.
• Analyze test failures identify root causes report defects and collaborate with engineering teams to resolve issues.
• Develop test datasets edge-case scenarios negative test cases and adversarial test scenarios for AI systems.
• Monitor test results maintain test documentation and continuously improve automation coverage and testing efficiency.

Required Technical Skills

• Strong programming experience in Python and object-oriented programming.
• Hands-on experience with test automation frameworks such as Pytest Selenium Playwright or Robot Framework.
• Experience with REST APIs JSON HTTP protocols Postman and API automation.
• Knowledge of AI/ML concepts including model inference training and validation data evaluation metrics and model behavior.
• Experience testing Generative AI applications LLM-powered solutions or AI-based APIs.
• Familiarity with AI evaluation techniques including exact-match scoring semantic similarity groundedness relevance and hallucination detection.
• Understanding of RAG architecture embeddings vector databases and retrieval quality.
• Experience with SQL databases and test data management.
• Knowledge of Git Linux/Unix debugging and software development life cycle methodologies.
• Experience with CI/CD tools and automated build and deployment pipelines.
• Strong analytical problem-solving communication and collaboration skills.

Preferred Qualifications

• Experience with AI testing and evaluation tools such as Ragas DeepEval promptfoo or LangSmith.
• Familiarity with LLM APIs and frameworks such as OpenAI APIs Hugging Face LanEligible to workhain or LlamaIndex.
• Experience with machine learning libraries such as scikit-learn PyTorch or TensorFlow.
• Familiarity with cloud platforms such as AWS Microsoft Azure or Google Cloud.
• Experience with Docker Kubernetes distributed systems and microservices.
• Knowledge of AI safety responsible AI model bias evaluation privacy and security testing.
• Experience building reusable test utilities synthetic test data generators and automated quality dashboards.
• Familiarity with observability tools model monitoring and production AI evaluation.

Education and Experience

• Bachelors or Masters degree in Computer Science Software Engineering Information Technology or a related field or equivalent practical experience.
• Typically 38 years of experience in software quality engineering SDET test automation or software development with relevant experience in AI testing.
• Demonstrated ability to build and maintain automated testing solutions for complex software systems.
• Experience working in Agile development environments and cross-functional engineering teams.

Key Success Metrics

• Increased automated test coverage for AI-powered applications.
• Improved detection of AI-specific defects hallucinations and reliability issues.
• Reduced regression testing time and faster release cycles.
• Improved model evaluation consistency and reproducibility.
• Reliable performance scalability security and quality of AI applications in production.

Ideal Candidate Profile
The ideal candidate combines traditional SDET expertise with emerging AI testing capabilities. They should be comfortable writing Python automation code testing APIs and distributed systems designing AI evaluation datasets and assessing the quality and reliability of LLM-powered applications. The candidate should be proactive in identifying edge cases and capable of translating complex AI behaviors into measurable repeatable test scenarios.
Key skills to highlight on your resume

• Programming: Python SQL scripting
• Test automation: Pytest Selenium Playwright API automation
• AI testing: LLM evaluation hallucination testing prompt testing AI quality metrics
• AI frameworks: RAG LanEligible to workhain LlamaIndex Ragas DeepEval
• API and backend: REST APIs Postman microservices
• DevOps: Git Jenkins CI/CD Docker
• Testing types: Functional integration regression performance security and end-to-end testing