AI Prompt Engineer Jobs in Canada

Job summary

AI prompt engineers design, test, and evaluate prompts for LLMs to ensure accurate and safe outputs. They collaborate with engineering and AI/ML data operations to improve model behavior through prompt iteration and evaluation.

Job details

About The Role AI prompt engineers design, test, and evaluate prompts that steer LLMs toward accurate, safe, and useful outputs across production workflows. You will collaborate with engineering and AI/ML data operations to improve model behavior through prompt iteration, evaluation, and RLHF-adjacent feedback. What You Will Do Design system prompts, user prompts, and tool instructions for LLM-powered products (chat, search, summarization, extraction, coding assistants). Run prompt evaluation and QA evaluation using rubrics to measure correctness, groundedness, safety, and style adherence. Create and maintain datasets for prompt testing, including data labeling and preference data aligned to RLHF objectives. Analyze failures (hallucinations, instruction gaps, unsafe content) and propose prompt, policy, or data fixes. Write and enforce annotation guidelines; ensure training data quality and compliance. Support LLM training pipelines, regression testing, and model performance improvement. Contribute to content safety labeling, refusal behavior testing, and jailbreak/misuse resistance. Document prompt libraries, evaluation sets, and decision logs for reproducible experiments. Required Qualifications Mid-Senior experience building or optimizing prompts for LLM applications in production or rigorous evaluation settings. Strong writing and reasoning skills to translate product intent into precise instructions and constraints. Hands-on experience with rubric-based prompt evaluation and QA evaluation. Understanding of RLHF concepts (preference data, calibration, helpfulness/harmlessness tradeoffs). Ability to define measurable criteria for quality and safety under ambiguity. Preferred Qualifications Experience with NLP tasks (summarization, classification, NER, extraction) and RAG evaluation. Familiarity with content safety labeling, policy writing, and red-teaming. Experience building evaluation datasets and running QA programs. Basic scripting/analytics (spreadsheets, SQL, Python) for tracking and error analysis. Compensation Hourly rate: $30–$50/hr (BASE_SALARY). Remote-ready roles across Canada time zones.

What you’ll do

AI prompt engineers design, test, and evaluate prompts for LLMs to ensure accurate and safe outputs. They collaborate with engineering and AI/ML data operations to improve model behavior through prompt iteration and evaluation.

Requirements

Candidates should have mid-senior experience in building or optimizing prompts for LLM applications. Strong writing and reasoning skills, along with an understanding of RLHF concepts, are essential.

Listed skills

• Évaluation · Preferred
• SQL · Preferred
• Production · Preferred
• Organization · Preferred
• Training · Preferred
• Data analysis · Preferred
• iteration · Preferred
• safety · Preferred
• Time · Preferred
• Tracking · Preferred
• Python · Preferred
• Labeling · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• AI Prompt Engineering
• LLM Applications
• Prompt Evaluation
• QA Evaluation
• Rubric-Based Evaluation
• Data Labeling
• Content Safety
• NLP Tasks
• Scripting
• SQL
• Python
• Model Performance Improvement
• Annotation Guidelines
• Feedback Analysis
• Regression Testing
• Decision Logs

Job areas

• Technology
• Engineering
• Data & Analytics
• Software

More jobs from Rex.zone