QE Lead with AI Experience

Job summary

Lead quality engineering and testing for AI/ML systems, including designing prompt-and-response test sets and evaluating accuracy, groundedness, relevance, and hallucinations. Provide auditable test evidence, defect analytics, and go/no-go recommendations for production readiness.

Job details

Job Title: QE Lead with AI Experience Location: Toronto, ON Work Arrangement: Hybrid (2-3 days a week) Employment Type: Contract / Full Time Salary/Pay Rate: Open to right candidate. Domain: BFSI Application Deadline: Nov. 13th, 2026 SKILLS REQUIRED: This role is open for both full time and well as contract. 10+ years of experience in Quality Engineering, including 2+ years evaluating AI/ML systems. Strong expertise in Python, test automation, API testing, and integration testing. Experience designing prompt/response test sets and evaluating accuracy, groundedness, relevance, and hallucinations. Hands-on experience with regression, bias/safety, performance, penetration, and non-functional testing. Familiarity with GitHub Actions, GitHub Copilot, Claude Code, Copilot Studio, LangSmith, AppInsights, Log Analytics, and Dynatrace. Ability to provide auditable test evidence, defect analytics, and go/no-go recommendations for production readiness. EEOC Compliance: We are an equal opportunity employer, and all qualified applicants will receive consideration for employment. DISCLAIMER AI Usage Policy: Pacer Group uses AI to assist in screening applications. Final hiring decisions are made by human recruiters based on qualifications and experience.

What you’ll do

Lead quality engineering and testing for AI/ML systems, including designing prompt-and-response test sets and evaluating accuracy, groundedness, relevance, and hallucinations. Provide auditable test evidence, defect analytics, and go/no-go recommendations for production readiness.

Requirements

Requires 10+ years of quality engineering experience, including at least 2 years evaluating AI/ML systems, with strong Python, test automation, API testing, and integration testing expertise. Candidates should have hands-on experience with regression, bias and safety, performance, penetration, and non-functional testing, and familiarity with the listed development and monitoring tools.

Listed skills

• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Quality Engineering
• AI/ML Evaluation
• Python
• Test Automation
• API Testing
• Integration Testing
• Prompt and Response Test Design
• Accuracy Evaluation
• Groundedness Evaluation
• Relevance Evaluation
• Hallucination Testing
• Regression Testing
• Bias and Safety Testing
• Performance Testing
• Penetration Testing
• Non-Functional Testing

Job areas

• Technology
• Software
• Management & Leadership
• Finance & Accounting

More jobs from Pacer Group