15Rock
Toronto, ON · Posted Oct 10, 2026

Applied AI Engineer — Evaluation & Reliability

Full-timeHybrid1 position

We’re checking whether this posting is accepting applications.

Browse current jobs
Save

Share this job

Help the right person find this opportunity.

The job in 20 seconds

LocationToronto, ON
Employment typeFull-time
Apply byApr 7, 2027
Posting languageEnglish

Job summary

Build deterministic evaluation gates that block reports with defects, and develop checks for fact consistency, citation validity, and source integrity. Create golden-set and regression testing harnesses, then harden validated approaches for production with CI and traceable evidence.

Job details

Applied AI Engineer — Evaluation & Reliability The problem you'd own 15Rock is a financial intelligence firm. Our research goes in front of institutional investors and gets read line by line — by people looking for the one number that doesn't hold up. Every figure has to reconcile across every table and sentence; every claim has to trace to a live source. We build that research with a multi-agent pipeline on large language models. LLMs write well and count badly: left alone, they produce clean prose that contradicts the table three paragraphs up. Our answer isn't "prompt harder." It's to route every number through deterministic computation and gate every report behind checks that pass or fail — never a model grading its own work. You'd build that layer. What you'll build Deterministic evaluation gates that block a report when a defect is present — measured against thresholds set before the run, not a gut feel. A fact-reconciliation layer that verifies a figure agrees with itself everywhere it appears. Citation-integrity checks: every reference resolves, every quote is real, every source is live. Golden-set and regression harnesses that catch quality drift before it ships. The path from research to production: hardening a validated approach into the pipeline, behind CI, with the traces to prove it works. What we're looking for You've shipped LLM systems to production — not demos — where being wrong had a cost. You've built the evaluations that kept them honest: you think in precision and recall, and you define "good" before you measure it. Strong Python; you can own backend services and data pipelines yourself. You reason about the numbers, not just the tokens — enough quantitative or financial literacy to know when a figure is wrong, or the appetite to learn it fast. Self-directing. You'd rather write a one-page memo than wait for a ticket. Bonus: model-risk or evaluation-infrastructure work in a regulated setting; LangGraph, DuckDB, CI you owned; a finance or quantitative background. This role is not Prompt engineering, a pure research seat, or a place where "the model said so" is an answer. How we work A small, senior team with direct access to the founder and research lead. Async by default — we meet on blockers, not to read status aloud. We run on hypotheses and measurable outcomes. Details Downtown Toronto, on-site / hybrid. Must be authorized to work in Canada. Early-stage financial intelligence firm, recognized by the Globe and Mail Changemaker Award and selected for Google for Startups. To Apply: send a short note and something you've built where correctness mattered to Careers@15Rock.com . Tell us about a check you built that caught something a person would have missed. 15Rock is an equal-opportunity employer.

What you’ll do

Build deterministic evaluation gates that block reports with defects, and develop checks for fact consistency, citation validity, and source integrity. Create golden-set and regression testing harnesses, then harden validated approaches for production with CI and traceable evidence.

Requirements

Candidates should have shipped production LLM systems where errors carried real costs and built evaluations using defined quality measures such as precision and recall. Strong Python skills and the ability to own backend services and data pipelines are required, along with quantitative or financial literacy or the ability to learn it quickly.

Listed skills

  • Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

  • Python
  • LLM Production Systems
  • Evaluation Design
  • Precision and Recall
  • Deterministic Computation
  • Fact Reconciliation
  • Citation Integrity
  • Regression Testing
  • Backend Services
  • Data Pipelines
  • Quantitative Analysis
  • Financial Literacy
  • CI
  • LangGraph
  • DuckDB

Job areas

  • Technology
  • Software
  • Data & Analytics
  • Finance & Accounting
About the employer
15Rock

Find where your market opportunities are. Then build the intelligence around them.

More jobs from 15Rock

See all jobs from 15Rock
15Rock
15Rock
Toronto, ON

Principal, Client Solutions

Full-timeHybrid
$230K–370K/yrView job
15Rock
15Rock
Toronto, ON

Engagement Manager

Full-timeHybrid
$100K–120K/yrView job
15Rock
15Rock
Toronto, ON

Senior Engagement Manager

Full-timeHybrid
$100K–120K/yrView job
15Rock
15Rock
Toronto, ON

Senior Engagement Manager

Full-timeHybrid
$100K–120K/yrView job