Freelance Gen AI Testing QA (Senior)

Role: Freelance Gen AI Testing QA Evaluation Engineer (Senior)

Company Name : WillWare Technologies

Work Model : Remote / Contract / Fulltime

Experience : 5 Years

Work Location :Chennai/Bangalore/Kochi/Jaipur/Coimbatore/Remote

Job description:

Experience:
5-8 years in QA automation with 1-3 years in GenAI / API-based testing.

Key Responsibilities:

Develop and maintain automated evaluation pipelines.
Implement evaluation scripts using Python frameworks (e.g. DeepEval custom frameworks)
Integrate LLM/Chatbot APIs and agent workflows into evaluation pipelines
Execute dataset-driven evaluations and capture and process responses.
Support manual test scenario execution and validation
Assist in dataset creation and enrichment
Generate evaluation reports and logs
Debug and troubleshoot execution issues.
Enable CI/CD integration for continuous evaluation.

Key Skills :

Core GenAI Evaluation Skills:
Experience with evaluation frameworks (e.g. DeepEval or Arize)
Understanding of LLM-as-a-Judge (G-Eval) methodology
Strong prompt engineering and evaluation design skills
Experience in manual evaluation of LLM outputs.

Technical Skills:

Strong programming in Python
Experience in API testing and integration
Proficiency in JSON handling parsing and data processing
Automation framework development/integration.
Knowledge of logging reporting and debugging tools

Agent Manual Testing & Dataset Skills:

Experience in:
o Test scenario creation for GenAI use cases
o Manual validation of LLM responses (qualitative assessment)
o Dataset creation and curation
o Writing expected outputs or golden answers.
Ability to design edge cases negative scenarios and adversarial inputs (prompt injection jailbreaks)

Domain & QA Skills:

Strong foundation in software testing principles:
o Functional integration regression testing
Experience in test design defect tracking and reporting.
Strong analytical and problem-solving skills.
Conversational AI testing experience.
Understanding of AI agent behavior workflows and edge cases.

Required Skills:

QA EngineeringGen AI TestingGenAIAPI TestingPython