AI QA Engineer

About the role

We are a 15-person health-tech B2B platform running a serverless-first AWS stack (Lambda API Gateway RDS Amplify Cognito).

Were looking for a QA Engineer / SDET who writes test code owns test infrastructure and is fluent in using AI tooling to author and maintain tests as well as testing AI-powered features themselves. Youll be the person who turns we keep shipping regressions into a durable automated safety net.

This is a hands-on code-writing role. Youll work directly with a small engineering team and own QA strategy end-to-end.

What youll do

• Design and maintain end-to-end and regression test suites in Playwright (our current framework) including using codegen and AI-assisted authoring to accelerate coverage.
• Use AI tooling (LLM-based test generation self-healing selectors AI code assistants) to reduce the cost of writing and maintaining tests with judgment about where AI helps and where it introduces flakiness.
• Test LLM- and MCP-powered product features: build evaluation harnesses define quality bars for non-deterministic outputs and design guardrail/regression checks for natural-language and agent-driven workflows.
• Attack our three standing testing gaps: mock drift untested Lambda infrastructure and recurring regressions that ship without accompanying tests.
• Build post-deploy smoke tests and contract tests against our integration layer (HubSpot Dwolla Intercom PostHog Slack).
• Partner with engineering to enforce a Definition of Done that requires tests and embed QA into the sprint cadence rather than bolting it on at the end.

Must-have

• Strong JavaScript/TypeScript; you write and maintain test code not just test plans.
• Production experience with Playwright Cypress or equivalent modern E2E frameworks.
• Hands-on experience using AI tools in a real testing workflow (test generation maintenance or triage).
• Experience testing non-deterministic / AI-powered systems or a clear credible approach to how youd do it.
• Comfort in AWS serverless environments (Lambda API Gateway) and testing distributed integrations.
• Ability to own strategy independently in a small team.

Nice-to-have

• Experience with LLM evaluation frameworks or building eval/guardrail systems.
• MCP or agent-based tooling familiarity.
• Prior QA-in-a-startup experience where you built the function from scratch.
• Health-tech fintech or regulated-industry background.

Why this role

Youll own QA at a company that treats testing as an engineering problem not a checkbox with real autonomy a modern stack and a mandate to bring AI into the workflow deliberately.