Who you are

• You enjoy pushing the limits of what LLMs are capable of, and you have built high-quality evaluation resources to measure those capabilities (datasets, simulators, environments, etc.)
• You have a track record of developing new methods and/or data to evaluate LLMs, e.g. publications at top-tier conferences, popular benchmarks, etc
• You have deep experience building with and around LLMs, and you have built tools for analyzing and understanding their performance
• You have strong software engineering skills
• If some of the above doesn’t line up perfectly with your experience, we still encourage you to apply! If you want to work really hard on a glorious mission with teammates that want the same thing, Cohere is the place for you

What the job involves

• Evaluation is critical to making progress in scaling intelligence. As models continue to become superhuman in many real-world use cases, we must continue to develop new techniques to accurately measure our models' performance on frontier capabilities
• In this role, you are responsible for creating next-generation evaluation methods and scalable infrastructure to measure LLM progress
• Develop evaluation benchmarks, datasets, and environments for measuring the bleeding edge of model capabilities
• Conduct research to push the state-of-the-art in LLM evaluation methods, including training LLM judges; improving evaluation efficiency; and scalably building high-quality datasets
• Build scalable tools for investigating and understanding evaluation results that are used by all members of technical staff at Cohere, as well as leadership and our CEO
• Learn from and work with the best researchers and engineers in the field

Benefits

• We cover 100% of premiums across health, dental, vision & travel
• We offer RRSP, 401K, and Pension Scheme contributions
• We offer lunch and fitness studio credits and a quality time fund which can be spent on things like dog walking and laundry services
• We want everyone who contributes to our success to get a ‘piece of the pie’
• Work Remote & PTO
• We know that some people work better at home, so we are remote-first, and sure that all meetings, events, and opportunities are set up for our distributed team
• Every employee at Cohere gets 6 weeks of paid time off & US and Canadian federal holidays, and we provide unlimited sick days
• All new parents (including those who adopt or go through a surrogate journey) are eligible to receive 100% salary for 6 months in Canada, the US, and the UK
• The decision and journey to having kids is different for each person and family. Part of our commitment to diversity is being able to support the widest variety of these scenarios as we can so we offer financial support for egg freezing and IVF for those in the US, Canada and the UK
• Every employee at Cohere receives an annual $2,000 education fund to use as they please