PeoplePilot
Toronto, ON · Posted Sep 18, 2026

Senior Software Quality Engineer – AI

Full-timeOn-site1 position

Applications are currently unavailable for this posting.

Browse current jobs
Save

Share this job

Help the right person find this opportunity.

The job in 20 seconds

LocationToronto, ON
Employment typeFull-time
Experience levelSenior · 5+ years
Posting languageEnglish

Job summary

Develop and execute comprehensive quality strategies and scalable automation frameworks for AI-enabled software products. Evaluate AI responses for accuracy and reliability while integrating quality checks into CI/CD pipelines.

Job details

We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks. The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery. The position is full-time and onsite in Toronto, ON. Key Requirements 5+ years of experience in Quality Engineering, QA, SDET, or Test Automation. 5+ years of programming experience with Java, Python, Groovy, or similar languages. Strong experience building and maintaining test automation frameworks. Hands-on experience with Selenium, Playwright, Cucumber/Gherkin, or RestAssured. Experience with API, database, integration, regression, and microservices testing. Experience with JUnit, TestNG, ReadyAPI, qTest, and JMeter. Hands-on experience testing AI/ML, Generative AI, conversational AI, or LLM-based applications. Knowledge of AI quality concepts such as prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias, and response consistency. Experience with CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing. Strong analytical, problem-solving, communication, and documentation skills. Key Responsibilities Develop and execute comprehensive quality strategies, test plans, test cases, and test data. Design, develop, and maintain scalable test automation frameworks. Create automated tests for AI/ML and Generative AI features. Evaluate AI responses for accuracy, relevance, safety, hallucination, reliability, and latency. Test APIs, microservices, databases, integrations, data pipelines, and third-party services. Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage. Integrate automated testing and AI quality checks into CI/CD pipelines. Identify, document, prioritize, and track defects through resolution. Analyze test coverage, quality metrics, AI evaluation results, and production feedback. Collaborate with Development, DevOps, Product, and QE teams to improve product quality and release readiness.

What you’ll do

Develop and execute comprehensive quality strategies and scalable automation frameworks for AI-enabled software products. Evaluate AI responses for accuracy and reliability while integrating quality checks into CI/CD pipelines.

Requirements

Requires over 5 years of experience in Quality Engineering and programming with Java or Python. Must have hands-on experience testing AI/ML and Generative AI applications along with proficiency in automation tools like Selenium or Playwright.

Listed skills

  • Java · Preferred
  • Python · Preferred
  • CI/CD · Preferred
  • Agile · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

  • Quality Engineering
  • Test Automation
  • Java
  • Python
  • Groovy
  • Selenium
  • Playwright
  • Cucumber
  • RestAssured
  • API Testing
  • Microservices Testing
  • AI/ML Testing
  • Generative AI
  • LLM Evaluation
  • CI/CD
  • Agile

Job areas

  • Software
  • Technology
  • Data & Analytics
  • Engineering
About the employer
PeoplePilot

Your HR Co-Pilot for Smarter Decisions | AI-Powered Analytics Solutions Making Complex HR Analytics Simple and Fast

More jobs from PeoplePilot

See all jobs from PeoplePilot
PeoplePilot
PeoplePilot
Toronto, ON

AWS Architect

Full-timeOn-site
PeoplePilot
PeoplePilot
Montréal, QC

Quality Engineer / Lead – Aerospace Domain

Full-timeOn-site
PeoplePilot
PeoplePilot
Montréal, QC

Manager – Structures Design (Advanced Engineering)

Full-timeOn-site
PeoplePilot
PeoplePilot
Montréal, QC

RCCA Lead

Full-timeOn-site