AI Trainer Jobs in Canada

Job summary

Evaluate large language models through RLHF-style ranking and scoring, prompt evaluation, data labeling, and quality assurance checks. Validate training data, document edge cases and systematic errors, and contribute to rubrics, calibration, gold sets, and regression testing to improve model performance.

Job details

About The Role AI Trainer jobs in Canada focus on improving large language models through RLHF-style evaluations, prompt evaluation, data labeling, and QA evaluation across real AI/ML training pipelines. You will follow detailed annotation guidelines, verify training data quality, and provide structured feedback that improves helpfulness, correctness, and safety. What You’ll Do Perform RLHF evaluations (pairwise ranking, rubric-based scoring) and write clear rationales Execute prompt evaluation for instruction-following, factuality, and safety Label and validate datasets for NLP and content safety labeling Run QA evaluation checks (consistency, agreement, systematic error discovery) Document edge cases and build error taxonomies to drive model performance improvement Collaborate on rubrics, gold sets, calibration, and regression testing for model updates Skills And Qualifications Experience with structured evaluation and guideline-driven judgment Strong writing and documentation for rationales and edge-case notes Familiarity with RLHF, LLM evaluation, and prompt evaluation workflows Comfort with data labeling, QA evaluation, and training data quality processes Bonus: multilingual evaluation, NER/classification tasks, or multimodal evaluation How To Apply Browse roles on Rex.zone and apply with a resume highlighting RLHF, data labeling, QA evaluation, annotation guidelines compliance, and examples of model evaluation feedback. Hourly base pay range: $30–$50.

What you’ll do

Evaluate large language models through RLHF-style ranking and scoring, prompt evaluation, data labeling, and quality assurance checks. Validate training data, document edge cases and systematic errors, and contribute to rubrics, calibration, gold sets, and regression testing to improve model performance.

Requirements

Candidates should have experience making structured, guideline-driven evaluations and writing clear rationales and documentation. Familiarity with RLHF, LLM and prompt evaluation, data labeling, quality assurance, and training data quality processes is expected; multilingual, NER/classification, or multimodal evaluation experience is a bonus.

Listed skills

• Évaluation · Preferred
• Organization · Preferred
• Training · Preferred
• Quality assurance · Preferred
• Attention to detail · Preferred
• Compliance · Preferred
• safety · Preferred
• Documentation · Preferred
• Assurance · Preferred
• Labeling · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• RLHF Evaluation
• Large Language Model Evaluation
• Prompt Evaluation
• Data Labeling
• Quality Assurance Evaluation
• Annotation Guidelines
• Training Data Quality
• Pairwise Ranking
• Rubric-Based Scoring
• Instruction-Following Evaluation
• Factuality Evaluation
• Safety Evaluation
• Dataset Validation
• Error Taxonomy Development
• Calibration
• Regression Testing

Job areas

• Technology
• Data & Analytics
• Education

More jobs from Rex.zone