Quality Engineer

Job summary

Develop and execute end-to-end testing strategies for AI/ML and GenAI solutions, assessing accuracy, reliability, safety, security, and performance. Build automated tests and AI evaluation datasets, document auditable quality evidence, analyze defects, and provide deployment-readiness recommendations in collaboration with cross-functional teams.

Job details

Job Title: Quality Engineer - AI/GenAI Validation & Testing Location: Toronto, ON (Hybrid, 3-4 days onsite per week) Contract Duration: 12 Months (Strong possibility of extension) Overview We are seeking a highly experienced Quality Engineer specializing in Artificial Intelligence (AI), Machine Learning (ML), and Generative AI (GenAI) solution validation. This role will be responsible for defining and executing comprehensive testing strategies that ensure AI-powered applications, agents, and intelligent workflows are accurate, reliable, secure, performant, and production-ready. The ideal candidate will combine deep expertise in Quality Engineering with hands-on experience evaluating Large Language Models (LLMs), AI agents, AI-assisted workflows, and enterprise AI platforms. You will work closely with Engineering, Architecture, Security, Risk, and Product teams to provide evidence-based assessments and go/no-go recommendations for production deployment. Key Responsibilities Define, develop, and execute end-to-end testing strategies for AI/ML and Generative AI solutions. Validate the accuracy, reliability, safety, and performance of AI models, AI agents, and LLM-powered applications. Design and maintain automated test frameworks using Python and modern test automation tools. Create and manage AI-specific test datasets, prompts, evaluation criteria, and benchmark scenarios. Conduct testing focused on: Groundedness and response relevance Hallucination detection and mitigation Bias, fairness, and safety validation Regression testing API and integration testing Functional and non-functional testing Performance and scalability testing Security and penetration testing Evaluate retrieval-augmented generation (RAG) implementations, agentic workflows, and enterprise LLM integrations. Develop auditable test evidence and quality metrics to support governance, risk, compliance, and architecture review processes. Analyze defects, trends, and quality indicators, providing actionable recommendations for remediation. Collaborate with development, product, cybersecurity, privacy, and risk teams throughout the software delivery lifecycle. Produce detailed quality assessments and deployment readiness recommendations. Required Qualifications 10+ years of experience in Quality Assurance, Quality Engineering, or Software Testing. 2+ years of hands-on experience evaluating AI, Machine Learning, or Generative AI solutions. Strong programming and scripting skills in Python. Extensive experience with: Test automation frameworks, API testing, Integration testing, Regression testing, non-functional testing, Performance and load testing Experience designing prompts, test suites, and evaluation datasets for AI systems. Deep understanding of: LLM evaluation methodologies, Hallucination assessment, Response quality measurement, Groundedness and relevance testing, Bias and safety evaluation, Experience working within Agile delivery environments. Strong analytical, troubleshooting, and communication skills. Desired Attributes Strong attention to detail and quality mindset. Ability to assess complex AI systems and communicate risks effectively. Experience producing executive-ready quality reports and recommendations. Comfortable working with cross-functional stakeholders across engineering, architecture, security, privacy, and business teams. Proven ability to drive testing strategy for complex enterprise technology initiatives.

What you’ll do

Develop and execute end-to-end testing strategies for AI/ML and GenAI solutions, assessing accuracy, reliability, safety, security, and performance. Build automated tests and AI evaluation datasets, document auditable quality evidence, analyze defects, and provide deployment-readiness recommendations in collaboration with cross-functional teams.

Requirements

Requires 10+ years of quality assurance, quality engineering, or software testing experience, including at least 2 years evaluating AI, ML, or GenAI solutions, and strong Python skills. Candidates should have extensive test automation and API, integration, regression, non-functional, and performance testing experience, plus expertise in LLM evaluation, prompt and dataset design, groundedness, hallucinations, bias, and safety.

Listed skills

• Agile · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Quality Engineering
• AI/ML Testing
• Generative AI Validation
• Python
• Test Automation
• API Testing
• Integration Testing
• Regression Testing
• Performance Testing
• Load Testing
• LLM Evaluation
• Prompt Design
• RAG Testing
• Hallucination Assessment
• Bias and Safety Evaluation
• Agile

Job areas

• Technology
• Software
• Engineering
• Data & Analytics
• Security & Safety

More jobs from Hays