Senior Software Quality Engineer AI

How Youll Make An Impact

As a Senior Software Quality Engineer you will bridge traditional software quality engineering with cutting-edge AI quality practices. You will design and implement comprehensive testing strategies that ensure both the reliability of backend infrastructure and the quality of Generative AI features powering our agentic applications. This role combines deep technical expertise in automated testing & AI/ ML focused quality engineering enabling you to establish quality standards that span from microservices architecture to AI/ML models. Youll be instrumental in building the testing frameworks and quality processes that ensure our products are robust reliable and production-ready.

What You Will Do

• Design and execute comprehensive test plans for backend services APIs and microservices architectures.

• Partner with Backend Engineers AI Engineers and Product teams to understand requirements and identify quality risks early in the development cycle.

• Develop automated testing frameworks using tools like Pytest Playwright unittest and integration testing libraries.

• Build end-to-end test suites that validate the integration between backend systems AI agents databases & services.

• Advocate for quality-first practices influencing architectural decisions and embedding testing into the development lifecycle.

• Design and implement automated evaluation frameworks for Generative AI features including LLM / SLM model testing prompt/ output validation and behavioral assessment of agentic workflows.

• Develop quality metrics and evaluation methodologies for LLM-based applications assessing accuracy consistency reliability quality of AI/ML models.

• Create and maintain curated evaluation datasets and synthetic test data that cover edge cases adversarial scenarios and real-world variability.

• Leverage AI evaluation tools and frameworks & establish observability and monitoring requirements to quality assessment.

• Establish observability and monitoring requirements including structured logging metrics collection and tracing for both backend services and AI agent behavior.

• Implement quality gates and acceptance criteria and create test documentation including test plans automation architecture quality reports and runbooks.

• Build strategies for testing in cloud environments & containerized execution.

• Implement performance testing load testing and reliability testing for production backend services and AI inference pipelines.

• Define and champion quality standards best practices and testing methodologies for both traditional backend systems and AI applications.

• Conduct code reviews with a focus on testability quality patterns and maintainability.

• Mentor junior quality engineers sharing expertise in automation frameworks AI testing approaches and quality engineering principles.

• Communicate quality insights risk assessments and test results effectively to technical and non-technical stakeholders using tools like Jira and Confluence.

The Experience You Bring

• 4 years of experience in software quality engineering test automation or software development with strong QA focus.

• Deep expertise in test automation frameworks and tools such as pytest unittest Selenium Playwright etc.

• Strong programming skills in Python or Java with ability to write production-quality test code and automation frameworks.

• Proven track record building automated testing infrastructure for backend services and APIs.

• Experience with various testing methodologies: unit testing integration testing end-to-end testing contract testing performance testing and test-driven development (TDD).

• Strong understanding of CI/CD pipelines and experience integrating automated tests into deployment workflows.

• Comfortable working in Linux/Unix environments for test execution and debugging.

• Familiarity with testing AI/ML applications or LLM-based systems with understanding of unique quality challenges in non-deterministic systems.

• Knowledge of Generative AI concepts agentic frameworks (ReAct chain-of-thought tool use) and LLM capabilities/limitations.

• Exposure to agentic AI patterns and knowledge of how to test tool orchestration multi-turn conversations and RAG systems.

• Excellent written and verbal communication skills with ability to document findings clearly and comprehensively.

• Strong collaboration skills working across cross-functional teams in Agile/Scrum environments.

• Experience with collaboration tools like Jira Confluence and Git for team workflows.

• 4 years of professional experience in software quality engineering test automation or software development with strong testing focus

Education

• Bachelors degree in Computer Science Software Engineering or related field.

Arent sure if youre a match We know that impostor syndrome and the confidence gap can prevent us from meeting spectacular candidates so dont hesitate to apply; you could be the perfect fit!

Location

Vancouver (Hybrid)

Compensation and Benefits

Boomi is committed to fair and equitable compensation practices. Base compensation for this position in our Vancouver hub ranges from $89923 - $112404 CAD annually applicable bonus. Final compensation will be determined by various factors including the candidates knowledge skills experience and geographic location.