Machine Learning Engineer (Data & Evaluation Infrastructure)
Job Summary
Job Description
We are seeking a Machine Learning Engineer (MLE) to manage our post-training evaluation pipeline. The role involves building and scaling evaluation processes to assess model capabilities across various tasks, pinpointing areas of failure, and driving improvements. Key responsibilities include identifying tasks for evaluation, creating or curating test cases and measurement methods, implementing evaluations through objective verification, LLM judging, reward modeling, or human evaluation. You will also be responsible for expanding coverage, deeply analyzing failure cases, identifying solutions, and developing scalable and accessible internal evaluation presentation methods, such as GUIs or Slurm scripts.
Must have:
- Experience with evaluation frameworks
- Experience with automated and human evaluation
- Ability to build evaluation infrastructure from scratch
- Scale existing systems
Good to have:
- History of OSS contributions
2 skills required
Job Details
We’re looking for an MLE to own our post-training evaluation pipeline. You’ll build and scale evals depth and breadth that measure model capabilities across diverse tasks, identify failure modes, and drive model improvements.
Responsibilities:
- Identifying tasks for evaluation coverage
- Creating, curating, or generating test cases and ways to measure these tasks
- Implementing evaluation through objective output verification, LLM judge/reward modeling, human evaluation, or any tricks of the trade you may bring to the table
- Adding coverage and diving deep into analyzing what’s really gone wrong in failure cases
- Identifying ways to remedy failure cases
- Developing ways to present and make the evals scalable and accessible internally (e.g. light GUIs, scalable Slurm scripts, etc for running the evals)
Qualifications:
- Strong experience with evaluation frameworks
- Experience with both automated and human evaluation methodologies
- Ability to build evaluation infrastructure from scratch and scale existing systems
Preferred:
- History of OSS contributions
Similar Jobs
Looks like we're out of matches
Set up an alert and we'll send you similar jobs the moment they appear!
Similar Skill Jobs
Looks like we're out of matches
Set up an alert and we'll send you similar jobs the moment they appear!
Jobs in Worldwide
Looks like we're out of matches
Set up an alert and we'll send you similar jobs the moment they appear!
Data Analysis Jobs
Looks like we're out of matches
Set up an alert and we'll send you similar jobs the moment they appear!
