AI QA Engineer
Job Location:
Monthly Salary:
Posted on:
Vacancies:
Job Summary
Immediate joiners required !! 2 days WFO - Hinjewadi Phase 3
Fulcrum Digital is an agile and next-generation digital accelerating company providing digital transformation and technology services right from ideation to implementation. These services have applicability across a variety of industries including banking & financial services insurance retail higher education food healthcare and manufacturing.
- Design and execute test cases for LLM agents RAG pipelines agentic workflows and AI-assisted decision tools
- Validate AI outputs against ground truth using structured accuracy scoring (NAICS risk exposure flags hazard group mapping)
- Detect hallucinations reasoning gaps source fabrication and misattributions in model-generated content
- Run multi-model comparative testing across GPT Claude Gemini and Perplexity evaluating accuracy latency and output completeness
- Test prompt versions iteratively and track accuracy changes across prompt cycles
- Validate citation accuracy document ingestion pipelines and cross-document context handling
- Design edge case and negative tests for AI-specific failure modes content filter triggers tool call limits missing documents and incomplete synthesis
- Perform regression testing after model upgrades prompt changes and backend fixes and maintain structured QA sign-off in JIRA
What Makes This Role Different from Traditional QA
- You evaluate whether an AI is reasoning correctly not just whether the UI behaves as expected
- You build evaluation rubrics for non-deterministic outputs and apply LLM-as-a-Judge techniques to score quality at scale
- You treat every model or prompt change as a potential accuracy regression not just a functional one
- You understand that in live AI systems a passing test today does not guarantee a passing test tomorrow
Required Skillsets
- 78 years of QA experience with minimum 2 years in Generative AI / LLM-based projects
- Hands-on experience testing chatbots RAG systems or agentic AI pipelines
- Proven ability to perform ground truth validation and detect hallucinations and reasoning failures
- Familiarity with multi-model evaluation prompt-aware testing and JIRA-based defect reporting
Preferred Skillsets
- Background in insurance or regulated industries; exposure to underwriting or risk classification concepts
- Familiarity with Azure OpenAI AWS Bedrock or SharePoint-integrated AI environments
- Knowledge of AI governance content filtering and PII redaction validation
Required Skills:
Who are we Fulcrum Digital is an agile and next-generation digital accelerating company providing digital transformation and technology services right from ideation to implementation. We specialise in delivering complex digital commerce ecosystems to large and small customers through a team-oriented and collaborative work environment. These services have applicability across a variety of industries including banking & financial services insurance retail higher education food health care and manufacturing. Job Summary Reports on a day-to-day basis to the Solution architect. The individual in this role will act as an integration developer and will have a broad understanding technology with a focus on technology stacks including those involving CRM solutions online content servers and online & mobile web applications. The developer will be involved in integrating and implementing solutions including legacy platform modernization through the project life cycle and ongoing support. This position will focus on translating applications from legacy solutions to web-based solutions using web framework and technology. Requirements Bachelors Degree from an accredited college or university with a major in Computer Science Information Systems Engineering Business or other related scientific or technical discipline. Masters Degree preferred. At least 4 years experience in developing and consuming RESTful APIs. At least 5 years experience in Service Oriented Architecture/Service integration with Web Services REST and security integration. Skills / Experience / Responsibilities Design develop and implement complex solutions based on Framework 4.5 using C# and . Will work with internal stakeholders to learn the business processes operating systems and application architectures in order to ensure that solutions fit the current or proposed architectural environment. Translate applications requirements into web-based solutions using the recommended frameworks and technology. Design and develop reusable components accelerators and frameworks. Assist in the system integration of components. Implement Test Driven Development frameworks (NUnit xUnit etc.). Implement Design Pattern (Singleton Factory and MVC). Implement Data Structures. Assist in translating customer service marketing & financial business function needs into digitization and automation of technical solutions aligned with organizational goals and strategies. Be a strong team player and effectively participate in projects and daily operational activities
