Senior AI Test Automation Engineer – LLM & Agentic Systems

San Jose, CA - USA

Monthly Salary: Not Disclosed

Posted on: 4 hours ago

Vacancies: 1 Vacancy

Job Summary

Who we are

Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers.

Job Description:

Job Title : Senior AI Test Automation Engineer LLM & Agentic Systems

Job Type : W2/C2C

Experience : 10-15 Years

Location : San Jose California

Responsibilities:

Minimum 8 years of experience in software QA / test engineering.
Strong understanding of testing principles: functional regression integration performance.
Hands-on experience testing AI or ML systems such as LLM chatbots or recommendation engines.
Hands-on experience with AI and Machine Learning (ML) techniques to enhance test automation such as generating test cases predicting potential defects or updating test code for user interface changes.
Hands-on experience with test automation frameworks and tools like Selenium Playwright or Cucumber using scripting languages like Python Java or JavaScript and leveraging AI to improve productivity.
Familiarity with key concepts of multi-agentic systems agentic frameworks (e.g. LangChain) and AWS AgentCore.
Scripting skills in Python and experience with REST API testing.
Design and run intelligent test strategies for LLM and agent-based systems.
Combining classic QA and test automation skills with a deep understanding of agentic automation and conversational AI
Design and execute end-to-end test plans for UX / Uber orchestrator/ Sub Agents.
Build and maintain agentic AI test suites validating autonomous.
Develop test data and synthetic scenarios for edge cases ambiguous queries and multilingual requests.
Evaluate agent behavior through reasoning traces tool use decisions and memory usage.
Implement and improve automated testing frameworks leveraging AI for conversational flows APIs and integrations.
Collaborate with Product management / cross-functional teams to refine requirements and acceptance criteria.
Document defects test results and insights clearly for stakeholders.

Qualification:

Bachelors degree or equivalent combination of education and experience.

Who we are Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers. Job Description: Job Title : Senior AI Test Automation Engineer LLM & Agentic Systems Job Type : W2/C2C Experience : 10-15 ...