Senior Automation Engineer (Python) – Agentic AI & LLM Systems (Hybrid)

Senior Automation Engineer (Python) Agentic AI & LLM Systems

Location: Glasgow (Hybrid 3 days per week onsite)
Start Date: Immediate
Right to Work: Must have full UK working rights (no sponsorship available due to start dates)
Additional Requirement: Must have 3 years of UK address history due to immediate onboarding
Programme: AI / Agentic AI

Seniority: Senior Automation Engineer

Overview

We are seeking a Senior Automation Engineer / SDET with strong Python skills and handson experience testing AI systems including LLMdriven applications and agentic AI workflows. You will design and build a scalable automation frameworks implement modern AI testing strategies and ensure the quality and reliability of nextgeneration conversational and agentbased AI products.

This role is ideal for someone who is passionate about automation engineering and has a solid understanding of how AI/LLM systems behave in realworld scenarios.

Key Responsibilities

• Develop and maintain a Pythonbased automation framework for validating AI agents (chatbots voicebots agentic workflows).
• Design and execute automated tests for:
  • LLM responses
  • Conversational flows
  • Retrievalaugmented generation (RAG)
  • Endtoend agentic scenarios
• Apply modern AI testing strategies and evaluation methods (e.g. DeepEval RAGAS beneficial but not essential).
• Integrate automated tests into CI/CD pipelines (Jenkins GitHub Actions Azure Repos).
• Build and manage batch and scheduled executions using Docker Kubernetes cron jobs or similar tooling.
• Manage code repositories using Git including branching strategies version control and collaborative workflows.

  Collaborate closely with engineering AI/ML and product teams to ensure highquality delivery of AIpowered features.

Required Skills & Experience

• Strong Python development skills particularly for automation (PyTest preferred).
• Solid background in test automation and framework design.
• Experience testing or validating AI/LLMbased systems (agentic workflows conversational AI RAG pipelines etc.).
• Understanding of AI evaluation concepts familiarity with tools like DeepEval or RAGAS is a bonus not a requirement.
• Experience implementing CI/CD pipelines and automated batch executions.
• Strong understanding of source code management Git workflows and collaborative engineering practices.
• Ability to design structured scalable and maintainable automation solutions.

Preferred Qualities

• Quick learner with a proactive ownershipdriven mindset.
• Strong problemsolving skills and ability to work in fastmoving AI environments.
• Comfortable collaborating with crossfunctional teams (AI/ML engineering product).
• Passion for AI technologies and modern testing methodologies.

Remote Work :

Yes

Employment Type :

Contract