Machine Learning Engineer, GenAI Quality

1 Day ago • 3 Years + • $172,000 PA - $300,000 PA

Job Summary

Job Description

This role focuses on developing ML systems to automate data quality evaluation and generation using large language models. You will build scalable systems to assess quality across accuracy, instruction adherence, factuality, and reasoning — and design robust evaluation frameworks to ensure alignment with human standards. You will be deeply involved in the full lifecycle: from model design and fine-tuning, to prototyping, deployment, and monitoring. You will partner closely with engineering, research, and product teams to deliver cutting-edge solutions for both customers and internal GenAI data engines.
Must have:
  • 3+ years of experience designing, training, and deploying ML models
  • Strong background in NLP, LLMs, and deep learning frameworks
  • Experience building microservices and deploying ML pipelines
  • Practical knowledge of LLM fine-tuning and evaluation
  • Strong programming skills and a solid foundation in algorithms
Good to have:
  • Experience with post-training LLM techniques
  • Familiarity with data evaluation pipelines, dataset curation
  • Background in multimodal ML or model evaluation

Job Details

About Scale:

Scale’s Generative AI ML team develops models and services to power high-quality data generation and evaluation for the most advanced large language models on earth. We also conduct applied research on model supervision and algorithmic approaches that support frontier models for Scale’s applied-ML teams and the broader AI community. Scale is uniquely positioned at the center of the AI ecosystem as a leading provider of training and evaluation data, end-to-end ML lifecycle solutions, and frontier evaluations for public and private institutions.

About The Role:

This role focuses on developing ML systems to automate data quality evaluation and generation using large language models. You’ll build scalable systems to assess quality across accuracy, instruction adherence, factuality, and reasoning — and design robust evaluation frameworks to ensure alignment with human standards. This is one of the highest impact areas in the company and directly accelerates the development of aligned, performant foundation models.

You’ll be deeply involved in the full lifecycle: from model design and fine-tuning, to prototyping, deployment, and monitoring. You’ll partner closely with engineering, research, and product teams to deliver cutting-edge solutions for both customers and internal GenAI data engines — Scale’s fastest-growing business.

If you’re excited about combining human-machine evaluation, scaling high-quality training data, and shaping the next generation of foundation models, we’d love to hear from you.

You will:

  • Design, fine-tune, and evaluate large language models for structured quality evaluation and data generation tasks
  • Develop robust evaluation frameworks to assess performance across accuracy, instruction following, reasoning, and other critical dimensions
  • Build and maintain scalable ML services to automatically assess and generate high-quality training and evaluation data
  • Research and apply state-of-the-art techniques in LLM training, post-training alignment (e.g., instruction tuning, RLHF), and tool-augmented reasoning
  • Collaborate with research scientists, engineers, and product teams to integrate your work into production services used by top AI developers

Ideally you’d have:

  • 3+ years of experience designing, training, and deploying ML models in production environments
  • Strong background in NLP, LLMs, and deep learning frameworks like PyTorch, TensorFlow, or JAX
  • Experience building microservices and deploying ML pipelines in cloud environments (e.g., AWS or GCP)
  • Practical knowledge of LLM fine-tuning and evaluation for tasks like factuality, instruction adherence, and chain-of-thought reasoning
  • Strong programming skills (e.g., Python) and a solid foundation in algorithms and data structures
  • Strong communication skills and experience working cross-functionally

Nice to haves:

  • Experience with post-training LLM techniques (instruction tuning, RLHF, tool use, or agent-based reasoning)
  • Familiarity with data evaluation pipelines, dataset curation, or scalable annotation workflows
  • Background in multimodal ML or model evaluation across domains such as code or long-context generation

Similar Jobs

Zscaler - Staff Data Science Engineer

Zscaler

San Jose, California, United States (On-Site)
9 Hours ago
Granicus - Data Scientist 4

Granicus

Bengaluru, Karnataka, India (Hybrid)
6 Months ago
Attentive - Staff Site Reliability Engineer

Attentive

(Remote)
4 Months ago
IMC - Machine Learning Engineer

IMC

Sydney, New South Wales, Australia (On-Site)
23 Hours ago
ByteDance - Software Engineer, Model Inference

ByteDance

Seattle, Washington, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Google - Senior Software Engineer, Machine Learning (Recommendations, Rankings, and Predictions)

Google

Mountain View, California, United States (On-Site)
2 Days ago
NVIDIA - Senior Research Engineer for Reinforcement Learning

NVIDIA

Canada (On-Site)
2 Months ago
ByteDance - Software Engineer in Machine Learning Systems

ByteDance

San Jose, California, United States (On-Site)
6 Months ago
ByteDance - Backend Engineer (Model Inference), Machine Learning Systems

ByteDance

Singapore (On-Site)
6 Months ago
Google - Senior Software Engineer, SDLC, Gemini Code Assist

Google

Kirkland, Washington, United States (On-Site)
1 Week ago
ByteDance - Algorithm Engineer - Audio Understanding

ByteDance

Singapore (On-Site)
6 Months ago
Meta - Software Engineer, Machine Learning

Meta

Bellevue, Washington, United States (On-Site)
5 Months ago
GoMotive - Computer Vision Engineer

GoMotive

(Remote)
1 Day ago
Western Digital - Data Scientist

Western Digital

Prachin Buri, Thailand (On-Site)
4 Weeks ago
NVIDIA - Senior AI-HPC Storage Engineer

NVIDIA

Westford, Massachusetts, United States (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

Jobs in San Francisco, California, United States

Twitch - Senior Manager - Corporate Communications

Twitch

Irvine, California, United States (On-Site)
3 Months ago
Epic Games - Animation Lead

Epic Games

Cary, North Carolina, United States (On-Site)
10 Months ago
Super - Senior Full-Stack Software Engineer ( Remote! )

Super

Orlando, Florida, United States (Remote)
6 Months ago
Evolution - Studio Game Presenter (Customer Service Alternative)

Evolution

Fairfield, Connecticut, United States (On-Site)
10 Months ago
Meta - Software Engineer - Datacenter networking

Meta

New York, New York, United States (On-Site)
5 Months ago
Passive Logic - Weather Simulation Engineer

Passive Logic

Salt Lake City, Utah, United States (On-Site)
4 Months ago
Samsung Semiconductor - Principal, Emulation Lead

Samsung Semiconductor

San Jose, California, United States (Hybrid)
1 Month ago
Scientific Games  - Software Quality Assurance Tester III

Scientific Games

Oklahoma City, Oklahoma, United States (On-Site)
3 Weeks ago
Scopely - Senior Manager, Business Operations - Office of the CRO

Scopely

California, United States (Hybrid)
3 Months ago
Next Level Business Services - Data Architect

Next Level Business Services

Sunnyvale, California, United States (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Similar Category Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

About The Company

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

Doha, Doha Municipality, Qatar (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Scale AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug