Our culture lifts you upthere is no ego in the way. Our common purpose We all want to win for our customers. We aim to always be evolving dynamic and ambitious. We believe in the power of genuine connections. Each employee is a part of what makes us unique on the market: agile and dedicated.

Time Type:

Job Description :

Specialist QA Test Automation

POSITION SUMMARY

We arent looking for an all-knowing AI guru to define our entire organizational strategy from scratch. We are looking for a sharp curious Specialist QA Test Automation who can bridge the gap between strategic vision and day-to-day technical execution. Working closely with the Chapter Lead QA who provides the overarching AI QA strategyyou will be embedded within Cogecos AI squads to bring that strategy to life.

In this role you will move past standard pass/fail scripts to tackle the complex world of non-deterministic outputs. You will focus heavily on probabilistic testing curating golden datasets and leveraging cutting-edge LLM evaluation tools like Promptfoo. If you have a solid foundation in QA or software engineering a healthy obsession with automation and a restless drive to learn youll fit right in.

KEY RESPONSIBILITIES

Probabilistic Testing & Evaluation

• Embrace the Unpredictable: Implement and refine testing workflows tailored for AI powered solutions where outputs are probabilistic rather than deterministic.
• Regression & Drift Detection: Build guardrails to identify model drift prompt regression hallucination risks and safety violations before they hit production.

Dataset Curation & Tooling

• Golden Datasets: Own the creation maintenance and scaling of high-quality golden datasets used as the source of truth for benchmarking model changes.
• AI Eval Tooling: Champion programmatic evaluation tools such as Promptfoo DeepEval or Ragas to automate the assessment of prompt variants and model performance.

Strategy Execution & Collaboration

• Chapter Alignment: Partner closely with the Chapter Lead QA to translate high-level AI QA frameworks into actionable daily tasks for technical teams.
• Feedback Loop: Provide real-time feedback to the Chapter Lead on how the strategy is performing in the trenches recommending practical pivots based on real-world results.

Automation & CI/CD Integration

• Smart Pipelines: Collaborate with DevOps specialists to inject AI evaluation metrics directly into modern CI/CD pipelines ensuring automated gates handle fluid AI responses.
• Data Validation: Validate data pipelines feeding our models ensuring that data integrity is maintained from ingestion to inference.

ESSENTIAL QUALIFICATIONS

Academic Training & Mindset

• College diploma or Degree in Computer Science Engineering or a related discipline (or equivalent practical experience).
• The Right Mindset: A strong foundation in core QA principles combined with a passion for AI/ML. You dont need a PhD; you just need to be eager to learn how LLMs behave.

Work Experience & Technical Skills

• 3 years of experience in Quality Assurance Software Engineering or Data Engineering.
• Hands-on Scripting: Experience in Python or JavaScript/TypeScript (essential for configuring evaluation frameworks like Promptfoo and manipulating test data).
• Modern QA Ecosystem: Experience with test automation frameworks API testing and code repositories (Git).

STRONG ASSETS

• Exposure to Golden Datasets: creating maintenance and/or scaling of high-quality golden datasets used as the source of truth for benchmarking model changes.
• Expsosure to AI Eval Tooling: Champion programmatic evaluation tools such as Promptfoo DeepEval or Ragas to automate the assessment of prompt variants and model performance.

• CI/CD Familiarity: Experience working with pipeline tools (e.g. Bitbucket pipelines Google Cloud run GitHub Actions GitLab CI Jenkins).

SPECIFIC COMPETENCIES

Technical Execution & Curiosity

• Analytical Deep-Dives: Comfortable analyzing data inputs/outputs and debugging prompts to identify the root causes of unexpected AI behavior.
• Tool Adaptability: Ready to quickly learn test and implement new open-source AI evaluation and observability tooling as the ecosystem evolves.

Growth Mindset & Adaptability

• Comfortable with Ambiguity: Thrives in a high-uncertainty environment where standard QA blueprints dont always apply.
• Execution Focus: Excel at taking an established roadmap or framework and running with it troubleshooting technical blockers independently.

Collaboration & Communication

• Squad Partner: Able to communicate technical testing gaps clearly to both AI developers and product owners.
• Open to constructive criticism: Open to constructive feedback willing to experiment and values an environment where mistakes are treated as learning milestones.

Youll benefit from:

• Flexibility: Yes we think that what you do matters. At work and at home.
• Fun: We laugh a lot it makes every day brighter.
• Discounted services: We provide amazing services to our clients and youll get them at home because you deserve them.
• Rewarding Pay: Lets be honest everybody likes to make a good salary. We offer attractive compensation packages and it comes with a great culture.
• Benefits: Weve got you covered.
• Career Evolution: Join us and we will give you the tools to achieve your career goals!
• Technology: You have a passion for technology Excellent we do too. Here you will manage influence play create fix and shape the industry.

#LI-KN1

#LI-Hybrid

Location :

Company :

At Cogeco we know that different backgrounds perspectives and beliefs can bring critical value to our business. The strength of this diversity enhances our ability to imagine innovate and grow as a company. So we are committed to doing everything in our power to create a more diverse and inclusive world of belonging.

By creating a culture where all our colleagues can bring their best selves to work were doing our part to build a more equitable workplace and world. From professional development to personal safety Cogeco constantly strives to create an environment that welcomes and nurtures all. We make the health and well-being of our colleagues one of our highest priorities for we know engaged and appreciated employees equate to a better overall experience for our customers.

If you need any accommodations to apply or as part of the recruitment process please contact us confidentially at