Staff AI QA Engineer

WHAT YOULL DO:

Join the dynamic team at Kaseya as a Staff AI QA Engineer. Ensure the reliability safety and ethical integrity of our AI platform by crafting advanced testing strategies and automation frameworks tailored for adaptive large-scale agentic systems and complex integrations. Youll play a critical role in validating AI behavior securing data pipelines and enforcing quality standards across distributed systems. Your work will help ensure our intelligent agents operate with precision fairness and resilience in real-world environments.

WHAT WE ARE LOOKING FOR:

Were looking for a quality-focused engineer who can think critically about testing complex adaptive systems. Youre a strong communicator who can clearly articulate quality risks and testing strategies to both technical and non-technical stakeholders. You thrive in collaborative environments working closely with developers data scientists and product managers and you bring leadership in QA best practices mentoring and ethical testing.

THE SCHEDULE:
This position is 100% remote.

ESSENTIAL DUTIES AND RESPONSIBILITIES:

• Design and implement comprehensive test strategies for AI agents platform services and integrations
• Develop and maintain automated test frameworks for functional performance and security testing
• Validate AI model behavior including prompt outputs bias detection and explainability
• Conduct rigorous testing of distributed systems microservices and real-time data pipelines
• Ensure data quality and integrity across training inference and integration workflows
• Collaborate with engineering and research teams to define quality gates and ethical testing standards
• Drive continuous testing practices in agile development cycles

WHAT YOULL BRING:

• 57 years of experience in QA or Test Engineering
• Mastery of test automation frameworks (e.g. PyTest Playwright Selenium RestAssured)
• Strong API testing skills (manual and automated)
• Experience with performance load and high-volume stress testing (e.g. JMeter k6s Gatling)
• Deep understanding of AI/ML model evaluation metrics and methodologies
• Experience testing distributed systems and microservices architectures
• Proficiency in data quality validation for AI training and inference workflows
• Experience with contract testing for microservices
• Ability to develop test strategies for non-deterministic and adaptive behaviors in AI systems (e.g. prompt validation guardrails bias detection)
• Familiarity with AI model testing tools (e.g. CheckList Giskard Deepchecks)
• Experience with test planning execution and reporting in agile environments
• Bachelors degree in Computer Science Software Engineering or a related field
• Preferred: Certifications in QA Automation AI/ML Testing or Ethical AI (e.g. ISTQB Certified Ethical Emerging Technologist)
• Additional training in AI safety fairness or model validation is a plus

Nice to Have:

• Experience with AI red teaming and adversarial testing techniques
• Explainability testing and validation for XAI methods (e.g. LIME SHAP)
• Usability testing for AI-driven UIs and agent interactions
• Familiarity with data privacy testing techniques (e.g. differential privacy validation)
• Security testing for AI models (e.g. model inversion membership inference poisoning attacks)
• Experience testing AI ethics fairness and bias
• Knowledge of formal verification methods for AI
• Experience building synthetic data generation pipelines for testing