Senior Software Development Test Engineer AI

About Tekion:

Positively disrupting an industry that has not seen any innovation in over 50 years Tekion has challenged the paradigm with the first and fastest cloud-native automotive platform that includes the revolutionary Automotive Retail Cloud (ARC) for retailers Automotive Enterprise Cloud (AEC) for manufacturers and other large automotive enterprises and Automotive Partner Cloud (APC) for technology and industry partners. Tekion connects the entire spectrum of the automotive retail ecosystem through one seamless platform. The transformative platform uses cutting-edge technology big data machine learning and AI to seamlessly bring together OEMs retailers/dealers and consumers. With its highly configurable integration and greater customer engagement capabilities Tekion is enabling the best automotive retail experiences ever. Tekion employs close to 3000 people across North America Asia and Europe.

About the Role
We are looking for highly motivated Senior AI Software Development Engineers in Test (AISDET) to join Tekions ML Engineering team. Our team builds and ships the machine learning
models and AI agents that power the Automotive Retail Cloud (ARC) from Service AI Agents and F&I automation to the AI-powered Analytics Agent trained on real-time data spanning 1700 dealership rooftops.
In this role you will be responsible for building confidence in the quality correctness and reliability of the models and AI agents we deliver. You will work closely with ML Engineers Data
Scientists the AI Platform team Product Management and Applied AI teams to define robust validation strategies automate testing across the model lifecycle and ensure our AI features meet the highest quality bar before they reach dealers and their you are testing AI systems you will go beyond deterministic assertions: you will design
evaluation frameworks for non-deterministic model and agent outputs measure accuracy and consistency catch hallucinations and regressions and continuously expand coverage. You will also use AI and LLMs yourself to generate test cases synthetic data and automation scaffolding raising both the speed and the depth of quality engineering.


What Youll Do

• Develop a deep understanding of Tekions AI agents ML models business domains and the data that feeds them.
• Own end-to-end quality for the ML models AI agents and AI-powered features shipped by the ML Engineering team.
• Design and implement automated testing frameworks for model inference prompt pipelines agentic workflows RAG/retrieval systems and the APIs that serve them.
• Validate model correctness accuracy latency and behavioral consistency across model versions prompt changes and upgrades.
• Define and execute evaluation strategies for AI/LLM-powered capabilities by:
• Creating and curating evaluation datasets (evals) and ground-truth sets
• Measuring response accuracy relevance and consistency
• Identifying hallucinations unsafe outputs and edge cases
• Building LLM-as-judge and automated scoring pipelines
• Continuously improving evaluation coverage as models and agents evolve
• Use AI/LLMs to accelerate quality engineering generating test cases synthetic test dataand automation scaffolding.
• Build regression suites that detect quality drift when models prompts embeddings or training data change.
• Validate integration of ML models and agents into ARC products including end-to-end flows and fallback behavior.
• Drive root cause analysis for production model and behavioral issues and partner with ML Engineers on preventive improvements.
• Define quality metrics guardrails and automated monitoring to detect model degradation before it impacts customers.
• Champion quality and evaluation engineering best practices across the ML Engineering organization.

What Were Looking For

• 58 years in SDET quality engineering or software engineering with a strong track recordof building test automation frameworks.
• Strong programming skills in Python (preferred for ML tooling) and/or Java with the ability to write production-quality automation code.
• Hands-on experience testing data-intensive or ML/AI systems or a strong softwaretesting background with demonstrated ML/LLM fluency.
• Solid understanding of ML concepts: model training and inference evaluation metrics (precision/recall F1 etc.) and the non-deterministic nature of model outputs.
• Experience designing evaluation frameworks or working with eval datasets benchmarks or LLM-as-judge approaches.
• Familiarity with LLM/agent concepts prompting RAG embeddings vector search and generative failure modes (hallucination drift prompt sensitivity).
• Experience with CI/CD test orchestration and API/integration testing.
• Strong analytical and debugging skills; comfortable performing root cause analysis across model data and code layers.
• Excellent collaboration and communication skills to partner across ML data and product teams.

Nice to Have

• Experience with ML/eval tooling such as MLflow Weights & Biases LangSmith Ragas DeepEval or provider eval suites.
• Experience with cloud platforms (AWS/GCP/Azure) and containerized environments (Docker Kubernetes).
• Exposure to model observability drift detection and production ML monitoring.
• Experience testing agentic systems tool use or multi-step LLM workflows.
• Prior experience in a high-scale SaaS or data platform environment.

Tekion is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race religion color national origin gender (including pregnancy childbirth or related medical conditions) sexual orientation gender identity gender expression age status as a protected veteran status as an individual with a disability victim of violence or having a family member who is a victim of violence the intersectionality of two or more protected categories or other applicable legally protected characteristics.

For more information on our privacy practices please refer to our Applicant Privacy Notice here.