LawZero Logo

LawZero

Director, Evaluations

Posted 26 Days Ago
Be an Early Applicant
In-Office
Montréal, QC, CAN
Expert/Leader
In-Office
Montréal, QC, CAN
Expert/Leader
Leads LawZero’s independent Evaluations Team, defining evaluation strategy, building benchmarks, datasets, environments, tooling, and red-teaming programs for Scientist AI and frontier LLM guardrails. Oversees capability, safety, explainability, honesty, adversarial robustness, and causal-mechanism evaluations. Builds an 8–10-person team, scales evaluation infrastructure, communicates results through technical reports and publications, and represents LawZero with AI safety institutes, collaborators, funders, and policymakers.
The summary above was generated by AI

LawZero is a non-profit building safe-by-design AI systems. We’re building the Scientist AI, an advanced AI system designed from the ground up to be both highly capable and safe. As we develop both general‑purpose Scientist AI models and safety guardrails for frontier LLMs, we need rigorous, independent evaluation of every capability and safety claim we make. We are looking for a Director of Evaluations to build, lead, and grow LawZero’s Evaluations Team.

This is a foundational hire. You will define what world‑class evaluation looks like at LawZero, build the team and infrastructure to deliver it, and ensure that evaluations remain independent of the main research stream so that capability and safety claims can be trusted both internally and externally by the wider AI and AI safety community.


Key responsibilities

  • Define LawZero’s evaluations strategy and roadmap, prioritising what needs to be measured and when, in close coordination with both research and product teams.
  • Build up the Evaluations Team during your first 3–6 months, scaling to roughly 8–10 people across research, engineering, dataset and benchmark design, and red‑teaming.
  • Operate the team independently of the main research and product streams in order to avoid conflicts of interest, including designing novel benchmarks that can be applied apples‑to‑apples to evaluate both the Scientist AI and frontier LLMs.
  • Oversee the design and construction of new datasets, tasks, and virtual or interactive environments to measure performance of the Scientist AI across  capabilities, safety (including honesty and goal-directedness), explainability, causal mechanisms and detecting adversarial attacks.
  • Lead evaluation of the Scientist AI when deployed as a guardrail around frontier models, including its ability to comply with harm specifications, detect and block harmful responses, explain its decisions, and resist adversarial attacks such as jailbreaks, prompt injection, and data poisoning.
  • Establish and lead our automated and manual red‑teaming programmes, both in‑house and in partnership with external providers, to stress test the Scientist AI as a general‑purpose model and as a guardrail.
  • Lead the construction of internal tooling and infrastructure needed to run evaluations at scale, automating and standardizing the pipeline wherever possible.
  • As needed and where possible, directly support research and product streams with their own internal requirements w.r.t. evaluations and benchmarking to unblock and speed up.
  • Own LawZero’s public communication of evaluation results, including model and system cards, technical reports, peer‑reviewed publications and blog posts, to build trust with the wider AI safety community.
  • Represent LawZero externally on evaluations and AI safety measurement, including engagements with AI safety institutes, research collaborators, and grant funders.

Skills and qualifications

  • An advanced degree (MSc or higher) in machine learning, computer science, or a closely related field.
  • 10+ years of experience in machine learning, with at least 5 years in a leadership role building or scaling technical teams working on real-world ML products.
  • Hands‑on expertise in designing and running large‑scale evaluations of LLMs or other frontier ML systems across capabilities, safety, and adversarial robustness.
  • A track record of building evaluation datasets, benchmarks, or interactive environments from scratch, including for safety‑relevant properties such as honesty, sycophancy, refusal behaviour, and adversarial robustness.
  • Strong written and verbal communication skills, including the ability to translate technical results for non‑technical audiences such as executives, funders, and policymakers.
  • Comfortable operating in a research‑driven, fast‑moving environment with significant ambiguity, and able to bring structure to it without slowing it down.

Nice to have: 

  • Experience leading red‑teaming exercises (automated, manual, or both) and working with third‑party evaluation or red‑teaming partners is a bonus.
  • Experience working with third-party partners for benchmark and dataset creation 
  • Experience releasing open‑source datasets, benchmarks, or evaluation tooling is a bonus.
  • Familiarity with current AI safety policy and standards work (UK AISI, CAISI, NIST, EU AI Act, etc.) is a bonus.
  • Experience contributing to or coordinating with external safety institutes, grant funders, or government bodies is a bonus.

What we offer

  • The opportunity to contribute to a unique mission with a major impact
  • Comprehensive health benefits
  • A minimum of 20 days vacation per year upon start
  • A minimum retirement savings employer contribution of 4%
  • Generous flexible benefits designed to contribute to your well-being
  • A team of passionate experts in their field
  • A collaborative and inclusive work environment with offices in the heart of Little Italy, in the trendy Mile-Ex district, close to public transportation.

About LawZero

LawZero is a non-profit organization committed to advancing research and creating technical solutions that enable safe-by-design AI systems. Its scientific direction is based on new research and methods proposed by Professor Yoshua Bengio, the most cited AI researcher in the world. Based in Montreal, LawZero’s research aims to build AI that learns primarily to understand the world rather than to act in it, giving truthful answers to questions based on transparent and externalized probabilistic reasoning. Such AI systems could be used to accelerate scientific discovery, to provide oversight for agentic AI systems, and to advance the understanding of AI risks and how to avoid them. LawZero believes that AI should be cultivated as a global public good—developed and used safely towards human flourishing. For more information, visit www.lawzero.org

You belong here

At LawZero, diversity is important to us. We value a work environment that is fair, open and respectful of differences. We welcome applications from highly qualified individuals interested in working towards our mission in a respectful, inclusive and collaborative setting.

Your personal information will be collected and processed by LawZero to evaluate your application for employment in compliance with our Privacy Policy. Under privacy laws in force in your country of residence, you may have several privacy rights, such as to request access to your personal information or to request that your personal information be rectified or erased. Details on how you can exercise your rights can be found in our Privacy Policy.

Similar Jobs

Senior level
Consumer Web • eCommerce • Mobile • Retail • Social Media • Software
Manage a portfolio of DACH sellers, developing growth strategies, analyzing performance, increasing platform adoption, and improving operational health. Build trusted seller relationships, provide coaching, advocate for seller needs, and collaborate with Product, Marketing, Category, and Policy teams. Lead concurrent initiatives, translate insights into business cases, execute marketing and operational improvements, and share partnership best practices across the team.
Top Skills: ChatgptClaudeGongGong AiHexSalesforceSalesforce AutomationSigma
Yesterday
In-Office
Expert/Leader
Expert/Leader
Consumer Web • eCommerce • Mobile • Retail • Social Media • Software
Own and scale marketing-led demand growth in France, driving buyer acquisition, repeat engagement, brand development, and creator ecosystem growth. Lead cross-channel initiatives across influencer marketing, paid media, partnerships, content, and events. Manage France-focused marketers, coordinate with category and creative teams, prioritize high-impact opportunities, and ensure measurable business results. The role requires strong knowledge of French consumer behavior, bilingual French and English fluency, and experience operating strategically and hands-on in ambiguous environments.
Yesterday
In-Office
Expert/Leader
Expert/Leader
Consumer Web • eCommerce • Mobile • Retail • Social Media • Software
Lead Whatnot’s French market launch strategy, including identifying priority categories, building zero-to-one launch playbooks, acquiring sellers, activating communities, and driving growth. Manage and develop a cross-functional launch team while partnering with product, marketing, operations, engineering, and executives. Represent the French market internally and externally, use data to guide experiments and decisions, and build relationships with sellers, buyers, and industry communities.

What you need to know about the Montreal Tech Scene

With roots dating back to 1642, Montreal is often recognized for its French-inspired architecture and cobblestone streets lined with traditional shops and cafés. But what truly sets the city apart is how it blends its rich tradition with a modern edge, reflected in its evolving skyline and fast-growing tech industry. According to economic promotion agency Montréal International, the city ranks among the top in North America to invest in artificial intelligence, making it le spot idéal for job seekers who want the best of both worlds.

Key Facts About Montreal Tech

  • Number of Tech Workers: 255,000+ (2024, Tourisme Montréal)
  • Major Tech Employers: SAP, Google, Microsoft, Cisco
  • Key Industries: Artificial intelligence, machine learning, cybersecurity, cloud computing, web development
  • Funding Landscape: $1.47 billion in venture capital funding in 2024 (BetaKit)
  • Notable Investors: CIBC Innovation Banking, BDC Capital, Investissement Québec, Fonds de solidarité FTQ
  • Research Centers and Universities: McGill University, Université de Montréal, Concordia University, Mila Quebec, ÉTS Montréal

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account