Senior QA Engineer, AI Agents & Agentic Workflows

QAD is seeking a Sr QA Engineer to own the continuous validation and quality of Champion Assist a multi-
agent AI product with 20 evolving agents. You will test the agents themselves: their reasoning tool use
orchestration and handoffs and the quality of their responses. This goes beyond UI and API checks. You will
design evaluation frameworks and regression suites that catch failures unique to non-deterministic LLM-
driven systems.
 

Responsibilities:

Design run and maintain end-to-end business scenarios that exercise multi-step agentic workflows
including agent routing tool/function calls and multi-turn conversations.

Evaluate agent outputs for correctness relevance groundedness consistency tone and
completeness using both human review and automated methods.

Build and maintain eval datasets (golden sets edge cases adversarial prompts) and automated
regression suites that run as agents prompts models and tools change.

Identify cross-agent orchestration and integration issues: wrong agent selection broken handoffs
tool-call failures context loss looping or runaway behavior.

Test for hallucination prompt injection jailbreaks data leakage and unsafe or out-of-scope responses.

Detect regressions caused by model upgrades prompt changes or retrieval/knowledge changes and
quantify them with metrics.

Triage and root-cause issues with Engineering and Product separating prompt model retrieval tool
and orchestration causes.

Re-test fixes validate the end-user experience and keep evals current as the product evolves.

Document test strategy eval results and quality trends clearly for global stakeholders.

Qualifications :

Bachelors degree in software engineering computer science or a related field.

3-5 years of QA experience with at least 1-2 years testing LLM-based applications AI agents or
agentic workflows.

Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria rubrics and
acceptance thresholds for AI responses.

Experience building or maintaining AI eval or regression suites (e.g. LLM-as-judge golden datasets
scoring rubrics).

Understanding of agentic concepts: tool/function calling multi-agent orchestration planning
memory/context RAG.

Experience with test automation tools (Selenium Playwright TestComplete or similar) and API testing.

Additional Information :

• Your health and well being are important to us at QAD. We provide programs that help you strike a healthy work-life balance.
• Opportunity to join a growing business launching into its next phase of expansion and transformation.
• Collaborative culture of smart and hard-working people who support one another to get the job done.
• An atmosphere of growth and opportunity where idea-sharing is always prioritized over level or hierarchy.
• Compensation packages based on experience and desired skill set

About QAD:

QAD  Redzone is redefining manufacturing and supply chains through its intelligent adaptive platform that connects people processes and data into a single System of Action. With three core pillars Redzone (frontline empowerment) Adaptive Applications (the intelligent backbone) and Champion AI (Agentic AI for manufacturing) QAD Redzone helps manufacturers operate with Champion Pace achieving measurable productivity resilience and growth in just 90 days.

QAD is committed to ensuring that every employee feels they work in an environment that values their contributions respects their unique perspectives and provides opportunities for growth regardless of background. QADs DEI program is driving higher levels of diversity equity and inclusion so that employees can bring their whole self to work.

We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race color sex age national origin religion sexual orientation gender identity status as a veteran and basis of disability or any other federal state or local protected class. 

Remote Work :

No

Employment Type :

Full-time