AI Chatbot Validation Quality Engineer

Job summary

Validate chatbot responses for accuracy, relevance, completeness, and consistency; create and execute test scenarios, cases, and regression suites; and identify hallucinations, defects, formatting issues, and knowledge gaps. Analyze conversations, maintain test automation, collaborate with development and AI teams to resolve defects, report quality metrics, and support UAT, release validation, and production monitoring.

Job details

Pay Range: Cad 40-45/hr One day onsite in a week Validate chatbot responses for accuracy| relevance| completeness| and consistency.Create and execute test scenarios| test cases| and regression test suites.Identify hallucinations| incorrect responses| formatting issues| and knowledge gaps.Perform chatbot conversation analysis and recommend quality improvements.Develop and maintain test automation for chatbot validation where possible.Work closely with developers| AI engineers| and content teams to resolve defects.Track| document| and report defects| test results| and quality metrics.Support UAT| release validation| and production quality monitoring.Key Skillsets5 Yrs of strong QA SDET Test Automation experience.Experience testing AI| LLM| GenAI| chatbots| virtual assistants| or conversational platforms.Knowledge of prompt engineering conceptsExperience with Jira| Confluence| GitHub| and Agile delivery.Experience with API testing and automation frameworks.Strong analytical and troubleshooting skills.Experience with Python| Selenium| Playwright| PyTest| or similar tools.Strong communication and documentation skills.

What you’ll do

Validate chatbot responses for accuracy, relevance, completeness, and consistency; create and execute test scenarios, cases, and regression suites; and identify hallucinations, defects, formatting issues, and knowledge gaps. Analyze conversations, maintain test automation, collaborate with development and AI teams to resolve defects, report quality metrics, and support UAT, release validation, and production monitoring.

Requirements

Requires five years of QA, SDET, or test automation experience, including experience testing AI, LLM, generative AI, chatbot, virtual assistant, or conversational platforms. Candidates should know prompt engineering, API testing, automation frameworks, Agile delivery, and tools such as Python, Selenium, Playwright, PyTest, Jira, Confluence, and GitHub, with strong analytical, troubleshooting, communication, and documentation skills.

Listed skills

• GitHub · Preferred
• Quality assurance · Preferred
• Jira · Preferred
• prompt engineering · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Quality Assurance
• SDET
• Test Automation
• AI and LLM Testing
• Chatbot Validation
• Test Scenario Design
• Regression Testing
• Prompt Engineering
• API Testing
• Python
• Selenium
• Playwright
• PyTest
• Jira
• Confluence
• GitHub

Job areas

• Technology
• Software
• Engineering

More jobs from LanceSoft, Inc.