PeoplePilotVerified Job Source

Senior Software Quality Engineer – AI

Toronto, ON
On-site
Posted Sep 18, 2026
1 position

Opens an external site

Employment type: Full-time
Experience level: Senior · 5+ years
Apply by: Oct 15, 2026
Posting language: English
Working hours: 40 hours per week
Seniority: Mid-Senior level

Job summary

Develop and execute comprehensive quality strategies and scalable automation frameworks for AI-enabled software products. Evaluate AI responses for accuracy and reliability while integrating quality checks into CI/CD pipelines.

Job details

We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks. The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery. The position is full-time and onsite in Toronto, ON. Key Requirements 5+ years of experience in Quality Engineering, QA, SDET, or Test Automation. 5+ years of programming experience with Java, Python, Groovy, or similar languages. Strong experience building and maintaining test automation frameworks. Hands-on experience with Selenium, Playwright, Cucumber/Gherkin, or RestAssured. Experience with API, database, integration, regression, and microservices testing. Experience with JUnit, TestNG, ReadyAPI, qTest, and JMeter. Hands-on experience testing AI/ML, Generative AI, conversational AI, or LLM-based applications. Knowledge of AI quality concepts such as prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias, and response consistency. Experience with CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing. Strong analytical, problem-solving, communication, and documentation skills. Key Responsibilities Develop and execute comprehensive quality strategies, test plans, test cases, and test data. Design, develop, and maintain scalable test automation frameworks. Create automated tests for AI/ML and Generative AI features. Evaluate AI responses for accuracy, relevance, safety, hallucination, reliability, and latency. Test APIs, microservices, databases, integrations, data pipelines, and third-party services. Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage. Integrate automated testing and AI quality checks into CI/CD pipelines. Identify, document, prioritize, and track defects through resolution. Analyze test coverage, quality metrics, AI evaluation results, and production feedback. Collaborate with Development, DevOps, Product, and QE teams to improve product quality and release readiness.

What you’ll do

Requirements

Requires over 5 years of experience in Quality Engineering and programming with Java or Python. Must have hands-on experience testing AI/ML and Generative AI applications along with proficiency in automation tools like Selenium or Playwright.