AI Quality Analyst

Ottawa - Canada

Monthly Salary: $ 45000 - 55000

Posted on: 2 hours ago

Vacancies: 1 Vacancy

Department:

Engineering

Job Summary

Our Approach to AI & Engineering

At Agility PR Solutions we use AI to scale high-quality work not to replace human judgment. Our internal AI agents support teams across the business and we believe the best results come from pairing automation with thoughtful human evaluation.

This role is designed for someone who is curious detail-oriented and interested in how AI systems improve over time. You enjoy reviewing work criticallyidentifyingpatterns and contributing structured feedback that helps technology teams iterate faster and more responsibly.

How Youll Make an Impact

Wereseeking an AI Quality Analyst to support the evaluation and continuous improvement of internal AI agents used across our platform. This is a junior hands-on role within the Engineeringdepartment focused on acting as a key human-in-the-loop for AI development.

Youllreview AI-generated outputsidentifyinaccuracies or inconsistencies and document clear actionable feedback that helps Engineering improve model accuracy reliability andconsistency. Your work will directly support faster iteration cycles and higher-quality automation across the business.

What Youll Get to Do

Review and evaluate outputs produced by internal AI agents across a range of use cases.
Assess whether AI-generated responses areaccurate consistent and supported byappropriate sourcesor logic.
Identifypatterns in errors inconsistencies or hallucinations including repeated phrasing or recurring failure modes.
Document findings clearly and consistently including what was corrected why a change wasrequired and any observed trends.
Provide structured actionable feedback to Engineering to support model improvement and training.
Partner with Engineering and QA teams to refine evaluation processes and quality standards.
Contribute to the development of lightweight guidelines and best practices for AI quality review.
SupportadditionalAI agent initiatives as new use cases are introduced across the organization.
Other duties asrequired.

Why Youll Bring

Early-career professional experience in areas such as technical support quality assurance data quality operations analysis or systems support.
Strong attention to detail and comfort reviewing large volumes of information.
Ability toidentifypatterns explain why an output is incorrect and articulate feedback clearly.
Comfort learning new tools platforms and internal systems quickly.
Clear written communication skills particularly for documentation and structured feedback.
Genuine interest in AI automation and the role humans play in evaluating and improving intelligent systems.

Nice to Have

Experience working with AI-powered tools or automated systems.
Familiarity with QA methodologies evaluation frameworks or quality review processes.
Exposure to SaaS products or internal tooling environments.

WhyYoullLove It Here

Fully remote flexible work environment.
Collaborative supportive engineering culture.
Health dental and vision benefits.
RRSP matching program.
Employee Assistance Program (EAP).
Career development and learning opportunities.
Paid vacation and sick days.
Summer Flex Fridays and a company-wide break between Christmas and New Years.
No internal meetings on Fridays.

Compensation for this role is expected to fall within the range of $45000$55000 CAD offers will reflect experience skills and internal equity.

This is for a vacant position.

AI Usage

AI is central to this role and you will work closely with AI-generated outputson a daily basis. AI tools are used to support efficiency and scale while humansremainresponsible for evaluation judgment and improvement.

AI does not make decisions at Agility. All decisions are made by people applying sound judgment andequitablepractices.

We believe technology should amplify humanexpertise not replace it.

At Agility we strive every day to build a more inclusive work environment and introduce tools that enable it. We strongly encourage applications from all people regardless of race religion gender age disability status or sexual orientation.

Even if you find yourself not checking all the boxes of our listed requirements but you are excited about this opportunity wed love to hear from you anyway. Our Talent Acquisition team will review your application for this role and other potential role matches open now or in the future.

If at any stage of the application process you require accommodation owing to disability or a medical need please let us know atfor us to make appropriate arrangements.

Thank you for your time in reviewing this opportunity and we hope to hear from you should you find this the right fit!

Required Experience:

Our Approach to AI & Engineering At Agility PR Solutions we use AI to scale high-quality work not to replace human judgment. Our internal AI agents support teams across the business and we believe the best results come from pairing automation with thoughtful human evaluation.This role is designed fo...