Senior QA Automation Engineer (AI-First)

Job Description :

Founded by pioneering journalists as the first social media newswire Storyful was created out of the need to break the news faster and utilize social content to add context to reporting. Acquired by News Corp in 2013 Storyful has evolved into a premium service for media business leaders and investors.

Our goal is to help our partners understand the attitudes behaviors and emotions shaping the world. Powered by a unique ability to gather and streamline data from all corners of an increasingly complex information environment our teams deliver clarity in a world of confusion. Our mission is to dig deeper into the nuance inherent in digital media to establish context verify the truth and help our partners make sense of the world.

Storyful is building AI-native products on top of complex high-volume multi-source content. We are looking for a hands-on Senior QA Automation Engineer to raise the quality bar while increasing delivery speed through AI-augmented engineering.

This is a senior individual contributor role for someone who treats AI as a core part of their craft using coding agents LLMs and evaluation frameworks to scale quality faster than headcount.

You will design and build the automation evaluation and quality foundations that enable engineers to own quality while ensuring our products remain reliable explainable and trusted even as they become increasingly probabilistic.

You will work closely with software engineers data scientists product managers and SRE to embed quality into every stage of the development lifecycle.

What you will do

Design and build AI-augmented automation systems

• Build and evolve scalable automation frameworks across web UI APIs and data pipelines (Playwright / TypeScript)

• Use AI coding agents (e.g. Claude Code) to generate refactor and maintain test suites at scale

• Translate natural-language acceptance criteria into executable maintainable tests

• Define reusable testing patterns that enable engineers to own automation within their domains

• Continuously reduce flakiness using trace analysis observability and AI-assisted failure triage

Enable shift-left engineering-led quality

• Partner with engineers to embed testing early in the development lifecycle

• Define pragmatic risk-based test strategies across unit integration contract and E2E layers

• Establish clear definitions of done quality gates and release readiness standards

• Build tooling and frameworks that allow teams to move fast without relying on central QA

Build QA practices for AI/LLM-powered features

• Design evaluation pipelines for LLM-driven features using:

  • Golden datasets and regression harnesses

  • Semantic similarity scoring

  • LLM-as-a-judge evaluation patterns

• Detect and prevent issues such as prompt drift hallucinations retrieval failures and schema regressions

• Test agentic workflows end-to-end (tool use multi-step reasoning failure modes)

• Partner with data science to define measurable quality signals (precision recall grounding drift)

Integrate quality into CI/CD and delivery

• Embed automation and evaluation into CI/CD pipelines for fast reliable feedback

• Define and tune meaningful quality gates based on real metrics (not ceremony)

• Support release processes with high-confidence signals and rollback readiness

Drive debugging observability and defect excellence

• Investigate complex issues across distributed systems using logs traces and metrics

• Improve observability including LLM trace visibility (e.g. Langfuse)

• Establish strong defect triage categorisation and reporting practices

• Use data (leakage flake rate MTTR escape rate) to continuously improve quality

Own non-functional and AI-specific risk testing

• Embed security testing into automation (OWASP API security auth flows)

• Test AI-specific risks (prompt injection jailbreaks unsafe tool use data leakage)

• Collaborate on performance resilience and scalability testing where it matters most

What good looks like in this role

• Engineers own quality by default supported by strong frameworks and AI tooling

• Automation is fast reliable and trusted as a release signal

• LLM-powered features ship with measurable repeatable quality evaluation

• Regression risk is proactively managed across both deterministic and probabilistic systems

• Delivery speed increases without compromising trust or stability

• Quality becomes a competitive advantage not a bottleneck

What were looking for

• Proven experience as a Senior QA Automation Engineer / SDET in production environments

• Strong hands-on experience with Playwright (or equivalent) using TypeScript

• Demonstrated use of AI coding tools (e.g. Claude Code) as part of daily workflow

• Strong software engineering fundamentals and ability to write maintainable production-grade code

• Experience designing automation frameworks and test strategies end-to-end

• Deep understanding of CI/CD and integrating automation into delivery pipelines

• Strong debugging and observability skills across distributed systems

• Experience driving shift-left practices and enabling engineers to own quality

• Solid understanding of databases APIs and test data validation (SQL REST/GraphQL)

• Security-aware mindset (OWASP secure test design)

Particularly valuable experience

• Testing AI/LLM-powered systems (evaluation frameworks prompt regression drift detection)

• Experience with LLM evaluation patterns (LLM-as-a-judge semantic scoring golden sets)

• Exposure to RAG pipelines vector search or knowledge graph systems

• Familiarity with LLM observability and experimentation tools (e.g. Langfuse MLflow)

• Experience testing agentic systems and multi-step workflows

• Performance and load testing at scale

• Cloud experience (AWS/GCP) and test infrastructure design

• Experience in high-trust domains (media intelligence risk compliance)

Why this role matters

This role will define how Storyful delivers quality in an AI-native world.

As our products evolve from deterministic systems to probabilistic agent-driven platforms traditional QA approaches no longer scale. This role ensures we can move fast while maintaining trust building the foundations that allow AI-powered features to work reliably at production scale.

Equal Opportunity Employer

We are committed to creating an inclusive workplace that values diversity. All qualified applicants will receive consideration for employment without regard to race colour religion gender gender identity or expression sexual orientation age disability national origin or any other protected characteristic under applicable local laws.

Reasonable Accommodation

We are committed to providing reasonable accommodation for qualified individuals with disabilities in our job application and/or interview process. If you need assistance or accommodation in completing your application or participating in an interview due to a disability email us at Please put Reasonable Accommodation in the subject line and provide a brief description of the type of assistance you need. This inbox will not be monitored for application status updates.

Please refer to the privacy notice at the bottom of this page for submitting any data access deletion or other data subject rights requests where permitted under your local laws and regulations.

Job Category: