Quality Engineer (QE) / SDET – AI Chatbot Validation

Job summary

Validate AI chatbot responses for accuracy, relevance, completeness, and consistency, and identify hallucinations, formatting issues, and knowledge gaps. Build and maintain automated validation and regression frameworks, investigate defects with cross-functional teams, and support UAT, release validation, and production quality monitoring.

Job details

Job Title: Quality Engineer (QE) / SDET – AI Chatbot Validation Locations: ~Brampton, Ontario~Scarborough, Ontario~Etobicoke, Ontario~Guleph, Ontario~Hamilton, Ontario~Kitchener, Ontario~Milton, Ontario~North York, Ontario~Ontario Nes,~Oshawa, Ontario~ Long Term Contract Experience: 6–8 Years Work Model: Hybrid – 1 day onsite per week Job Description: We are looking for an experienced Quality Engineer / SDET with strong test automation skills and hands-on experience validating AI-powered chatbots, LLMs, Generative AI applications, or conversational platforms. The ideal candidate will ensure the accuracy, relevance, consistency, and reliability of AI-generated responses while developing and maintaining automated testing solutions. Key Responsibilities: Validate chatbot responses for accuracy, relevance, completeness, and consistency. Design, develop, and execute test scenarios, test cases, and regression test suites. Identify AI hallucinations, incorrect responses, formatting issues, and knowledge gaps. Analyze chatbot conversations and recommend quality improvements. Develop and maintain automated test frameworks for chatbot validation. Collaborate with developers, AI engineers, and content teams to investigate and resolve defects. Track defects, document test results, and report quality metrics. Support UAT, release validation, and production quality monitoring. Required Skills: 5+ years of strong QA / SDET / test automation experience. Experience testing AI, LLMs, Generative AI applications, chatbots, virtual assistants, or conversational platforms. Understanding of prompt engineering and AI response validation. Strong programming and automation skills using Python, Selenium, Playwright, PyTest, or similar tools. Experience with API testing and automation frameworks. Familiarity with Jira, Confluence, GitHub, and Agile delivery methodologies. Strong analytical, troubleshooting, communication, and documentation skills. Preferred Skills: Experience with LLM evaluation, hallucination detection, and conversational AI quality metrics. Experience building automated validation and regression testing frameworks for AI applications. Knowledge of Generative AI quality engineering practices.

What you’ll do

Validate AI chatbot responses for accuracy, relevance, completeness, and consistency, and identify hallucinations, formatting issues, and knowledge gaps. Build and maintain automated validation and regression frameworks, investigate defects with cross-functional teams, and support UAT, release validation, and production quality monitoring.

Requirements

Requires at least five years of QA, SDET, or test automation experience, including hands-on testing of AI, LLM, Generative AI, chatbot, or conversational applications. Candidates should have strong programming and automation skills, experience with API testing, familiarity with prompt engineering and tools such as Python, Selenium, Playwright, PyTest, Jira, Confluence, and GitHub, and strong analytical and communication skills.

Listed skills

• Quality assurance · Preferred
• Jira · Preferred
• Agile · Preferred
• prompt engineering · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Quality Assurance
• Software Development Test Engineering
• Test Automation
• AI Chatbot Validation
• Large Language Models
• Generative AI
• Prompt Engineering
• Python
• Selenium
• Playwright
• PyTest
• API Testing
• Hallucination Detection
• Regression Testing
• Jira
• Agile

Job areas

• Technology
• Software
• Engineering

More jobs from Realtek Consulting LLC