Opens an external site
- Employment type
- Contract
- Minimum education
- Bachelor’s degree
- Posting language
- English
Job summary
Conduct voice-based roleplay conversations with AI models to assess and compare their performance across predefined quality dimensions. Document findings, identify error patterns, and prepare detailed rationales to support evaluation results.
Job details
This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for an AI Voice Evaluation Specialist based in Canada. Join an innovative AI-focused environment as an AI Voice Evaluation Specialist, where you will contribute to improving the quality, reliability, and performance of next-generation conversational AI systems. In this role, you will interact directly with two different AI models through real-time voice conversations, using identical scenarios to assess and compare their performance. You will evaluate conversational quality, identify potential errors, and analyze how effectively each model responds to different situations. Your observations and structured assessments will help determine which systems deliver the strongest and most natural user experiences. This opportunity is ideal for someone with excellent listening skills, strong analytical abilities, and a keen interest in artificial intelligence and voice-based technologies. You will work independently within a structured evaluation framework, ensuring consistency, objectivity, and accuracy throughout each assessment. \n Accountabilities: Review assigned scenarios and conduct natural, voice-based roleplay conversations with two different AI models. Interact with Model A and Model B using the same scenarios, maintaining consistent conversational approaches and conditions to ensure fair comparisons. Keep approximately the same number of conversational turns with each model to support balanced and reliable evaluations. Record voice interactions and carefully assess the quality, accuracy, relevance, and overall behavior of each model's responses. Evaluate both AI models across five predefined quality dimensions, following established assessment criteria and evaluation guidelines. Identify, document, and categorize relevant error patterns and clusters occurring during audio interactions or conversations. Compare the performance of both models using structured evaluation frameworks and observations gathered throughout the interactions. Determine which model demonstrates stronger overall performance based on the defined quality standards and evaluation criteria. Prepare detailed written rationales supporting final evaluations, referencing specific conversational examples and observations from both models. Apply evaluation guidelines consistently across different scenarios and interactions to ensure objective, accurate, and dependable results. Maintain high standards of documentation, attention to detail, and evaluation consistency throughout all assigned tasks. Requirements: Bachelor's degree or equivalent relevant educational background. Excellent attention to detail, with the ability to identify subtle differences in conversational quality, response accuracy, and AI model behavior. Strong listening and comprehension skills, including the ability to understand spoken interactions and recognize inconsistencies or errors. Excellent written communication skills, with the ability to document findings clearly, objectively, and accurately. Ability to follow detailed instructions, structured evaluation frameworks, and predefined quality assessment guidelines. Confidence in speaking naturally and participating in roleplay conversations across different scenarios. Strong analytical and critical-thinking abilities, with the capacity to compare two interactions fairly and identify meaningful performance differences. Objectivity and impartiality when evaluating AI-generated responses, without allowing personal preferences to influence final assessments. High levels of reliability, consistency, and time management when completing repetitive and structured evaluation assignments. Genuine interest in artificial intelligence, conversational AI assistants, voice technologies, or emerging AI applications. Ability to work independently while maintaining accuracy and quality throughout the evaluation process. Benefits: Competitive hourly compensation ranging from CAD $20 to $25 per hour, depending on experience, skills, and qualifications. Full-time contract opportunity for an initial period of 3 months. 40-hour working week. Remote working arrangement within Canada. Opportunity to contribute directly to the evaluation and improvement of advanced AI voice technologies. Hands-on exposure to conversational AI systems and emerging artificial intelligence applications. Opportunity to develop practical experience in AI model evaluation, voice interaction analysis, and structured quality assessment. Collaborative, technology-focused working environment with clearly defined evaluation processes and quality standards. \n How Jobgether works: We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best! Why Apply Through Jobgether? Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time. #LI-CL1
What you’ll do
Conduct voice-based roleplay conversations with AI models to assess and compare their performance across predefined quality dimensions. Document findings, identify error patterns, and prepare detailed rationales to support evaluation results.
Requirements
Requires a bachelor's degree or equivalent and strong analytical, listening, and written communication skills. Candidates must be able to work independently, follow structured evaluation frameworks, and maintain objectivity when assessing AI performance.
Benefits
- Competitive hourly compensation
- Remote working arrangement
- Hands-on exposure to conversational AI systems
- Practical experience in AI model evaluation
Listed skills
- Critical Thinking · Preferred
- Attention to detail · Preferred
- written communication · Preferred
- Documentation · Preferred
- Time management · Preferred
Other relevant skills
Identified from the job description. Confirm important requirements above.
- AI evaluation
- Voice interaction analysis
- Analytical skills
- Critical thinking
- Attention to detail
- Listening skills
- Written communication
- Roleplay
- Quality assessment
- Documentation
- Time management
- Objectivity
- Conversational AI
- Communication
- Influencing Skills
- Innovation
- Detail Oriented
- Data Privacy
- Artificial Intelligence
- General Data Protection Regulation (GDPR)
- Analytical Skills
- Information Privacy
- Time Management
- Critical Thinking
- Quality Management
- Applications Of Artificial Intelligence
- Quality Assessment
- Listening Skills
- Voice Interaction
Job areas
- Technology
- Data & Analytics
- Software
- Science & Research
- Customer Service & Support
- Evaluation Specialist
- Customer Service Representative (General)
- Inquiry Clerks
- Customer Service Representatives
More jobs from Jobgether
Senior Software Engineer – Developer Platform
- Remote
- Canada, Hidalgo del Parral, Chihuahua, Mexico
- Posted Nov 17, 2026
Senior Software Engineer, Data Solutions
- Remote
- Canada
- Posted Nov 17, 2026
Senior Product Security Engineer
- Remote
- Canada
- Posted Nov 17, 2026
Medicaid Provider Data Service Senior Coordinator
- Remote
- Canada
- Posted Nov 17, 2026
