Senior AI QA Engineer

Job Description:

We are seeking a detail-oriented QA Engineer to test and assure the quality performance reliability and compliance of solutions built using Microsoft Power Platform and Copilot Studio including data integrations and AI/ML-enabled features. This role will lead test planning and execution across low-code applications automated workflows APIs/custom connectors and conversational AI experiences partnering closely with developers data engineers/scientists and business stakeholders.

Key Responsibilities:

• Define and maintain QA strategy test approach and coverage for Power Platform and Copilot Studio solutions.
• Create review and execute test plans test cases and test scripts across SDLC phases.
• Perform functional smoke regression integration end-to-end testing.
• Ensure traceability between requirements test cases defects and releases.
• Validate Dataverse/Databricks tables/entities Power Automate flow relationships security roles and data integrity rules.
• Design and execute conversation test scenarios for multi-turn context ambiguity handling and error recovery.
• Perform regression testing for AI bot behaviour changes due to configuration updates connector changes or model/version updates.
• Perform API testing for REST services Azure ML endpoints and custom integrations using tools such as Postman/JMeter.
• Test data integrations and pipelines involving Azure Databricks Azure Storage and Dataverse.
• Plan and execute performance and load tests for workflows and APIs
• Use JMeter to simulate real-world traffic patterns and produce performance reports
• Identify document and track defects using Azure DevOps Jira or equivalent tools.
• Communicate test status risks and release readiness to stakeholders.
• Collaborate with developers data scientists and product owners to clarify acceptance criteria and non-functional requirements.
• Contribute to QA best practices reusable test assets and automation frameworks.

Required Qualifications:

• 3 years of QA/testing experience including 12 years testing Microsoft Power Platform solutions.
• Hands-on experience testing Power Automate workflows and data-centric applications.
• Working knowledge of Dataverse connectors/custom connectors Azure Databricks and Azure Storage.
• Familiarity with Copilot Studio or chatbot testing methodologies (NLU multi-turn flows fallback handling).
• Strong understanding of software testing principles (test design execution defect lifecycle reporting).
• Experience with API testing (REST) using Postman and/or scripted validation (Python preferred).
• Hands-on performance testing experience using JMeter or LoadRunner including analysis and reporting.
• Ability to validate AI/ML outputs and collaborate with data teams on statistical checks and model behaviour.
• Basic understanding of designing prompts and edge cases to evaluate AI agent behaviour.
• Develop and maintain automation scripts for API testing Regression test suites Chat flow validation
• Build automated validation for chatbot responses using Pattern matching Similarity scoring Semantic validation (where applicable)

Preferred Qualifications:

• Microsoft Certifications in Power Platform or Data Analytics/AI.
• Experience in cloud-based testing environments (Azure preferred).
• Knowledge of accessibility usability and security testing.
• Exposure to CI/CD pipelines and integration of performance testing.
• Agile/Scrum project experience with Azure DevOps pipelines for Power Platform or Copilot deployments.
• Understanding of RAI principles
• Understanding of LLM-based systems Prompt engineering basics RAG architecture Multi-agent workflows

Tools & Technologies:
Azure DevOps
Postman
Selenium (Python automation)