Senior Software Engineer, GenAI Model Evaluation

8 Months ago • 5 Years + • Software Development & Engineering

Job Summary

Job Description

Scale is seeking a Senior Software Engineer to build a world-class model evaluation platform for their GenAI Safety & Evaluation product team. This role involves owning large new areas within the product, working across backend, frontend, and LLMs, and collaborating with cross-functional teams. Must-have skills include proficiency in Python, Node, React, and MongoDB, along with experience in algorithms, data structures, and object-oriented programming.
Must have:
  • Python, Node, React
  • MongoDB experience
  • Algorithms, data structures
  • Object-oriented programming
Good to have:
  • AI platforms & technologies
  • Generative models & LLMs
  • ML infrastructure building
  • AI-powered solutions
Perks:
  • Hyper-growth startup
  • Work with AI technologies

Job Details

About Job

Software is eating the world, but AI is eating software. We live in unprecedented times – AI has the potential to exponentially augment human intelligence. As the world adjusts to this new reality, leading tech companies are racing to build LLMs at billion dollar scale, while large enterprises figure out how to add it to their products. To ensure that these models are safe, aligned, and highly useful, they require extremely high quality human-generated data and evaluation. Since before the launch of ChatGPT, through to the latest generation of frontier models coming out today, Scale has been at the forefront of providing the post-training, fine-tuning, and human preference alignment (RLHF) data needed to ensure these models are capable, aligned, and useful via our Generative AI Data Engine. The data we are producing is some of the most important work for how humanity will interact with AI.

As customers train their models on this data, and constantly aim to improve them, a critical need is having trustworthy evaluations of model performance, and an ability to identify weaknesses and potential vulnerabilities. Conducting these evaluations with our human experts constitutes a significant and growing portion of Scale’s work—thus assisting model developers in iteratively understanding where to focus their technical investments.

The GenAI Safety & Evaluation product team at Scale is at the heart of this work, building a world-class customer-facing model evaluation platform. This platform enables customers to easily launch new evaluation workflows, deep dive into evaluation results down to the test case level to understand weaknesses and benchmark performance, and use these insights to drive model development roadmaps. In building this product, you will have a chance to shape the way that models across the industry are evaluated, impacting billions of people around the world. And as a newer product at Scale, you will have the opportunity to build something impactful from the ground up.

As part of the Safety & Evaluation product team, you will partner closely with researchers from Scale’s Safety, Evaluations, and Alignment Lab (SEAL) on productization of novel research, as well as Scale’s expert red team, which supports AI safety via rigorous model testing trusted by the White House, major enterprises, and leading model developers.

 

We’re looking for entrepreneurial Software Engineers to join our team. In this role, you'll be given the opportunity to build these products and drive millions of dollars in revenue. You’ll also get widespread exposure to the forefront of the AI race as Scale sees it in enterprises, startups, governments, and large tech companies.

The ideal person is a natural entrepreneurial engineer who can take an ambiguous scope and lead the execution of outcomes, doing what it takes to hit them including both backend and frontend coding, defining requirements, coordinating with other eng and operations teams at Scale, etc. We strongly believe the best engineers own outcomes and deeply understand customer problems.

 

You will:

  • Own large new areas within our product, delivering customer-ready features with engineering excellence that stands up to rigorous quality standards
  • Work across backend, frontend, and interacting with LLMs and/or other ML models
  • Work across the entire product lifecycle from conceptualization through production
  • Be able, and willing, to multi-task and learn new technologies quickly
  • Collaborating with cross-functional teams to define, design, and ship new product features and experiences.
  • Be ready to jump in on fast-turnaround product requests for high value customers

 

Ideally you'd have:

  • 5+ years of full-time engineering experience, post-graduation
  • Proficiencies in one or more of Python, Node, React, Next.js and MongoDB
  • Solid background in algorithms, data structures, and object-oriented programming
  • Experience scaling products at hyper-growth startups
  • Excitement to work with AI technologies
  • Strong written and verbal communication skills, to be able to thrive in a writing-first culture
  • Strong problem-solving skills, and be able to work both independently and as part of a team

Nice to haves:

  • Strong knowledge of software engineering best practices
  • Have experience with AI platforms and technologies, including generative models and LLMs
  • Experience building ML infrastructure and AI-powered solutions
  • Experience growing new products from 0 to 1

Similar Jobs

Whatnot - Head of Marketing Strategy and Operations

Whatnot

Los Angeles, California, United States (On-Site)
1 Month ago
Stibo Systems - Director of Pricing & Monetization

Stibo Systems

Aarhus, Denmark (On-Site)
2 Months ago
bytedance - Shop Tab Strategist

bytedance

Taguig, Metro Manila, Philippines (On-Site)
3 Months ago
Toast - Talent Acquisition Partner, Contractor

Toast

United States (Remote)
2 Weeks ago
Epic Games - Marketing Director

Epic Games

London, England, United Kingdom (On-Site)
2 Months ago
Lytx,  Inc  - Senior Sales Engineer

Lytx, Inc

United States (Remote)
1 Day ago
PwC - Senior Associate SAP SD

PwC

Mumbai, Maharashtra, India (On-Site)
3 Weeks ago
Tesla - Electronics Failure Analysis Engineer EMEA

Tesla

North Brabant, Netherlands (On-Site)
4 Months ago
Alten Technology - Staff Embedded Software Engineer (AMD Kria)

Alten Technology

Lafayette, Colorado, United States (Hybrid)
1 Month ago
Haleon - Engineering Validation Specialist

Haleon

Panama City, Panamá Province, Panama (On-Site)
1 Year ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

bytedance - Research Scientist in Foundation Model, Speech Understanding - 2025 Start (PhD)

bytedance

San Jose, California, United States (On-Site)
8 Months ago
Penumbrainc - Sr. Quality Engineer

Penumbrainc

Roseville, California, United States (On-Site)
3 Months ago
Armada - Principal Engineer (Applications)

Armada

Thiruvananthapuram, Kerala, India (On-Site)
8 Months ago
Rackspace Technology - Azure Cloud Architect

Rackspace Technology

Gurugram, Haryana, India (Remote)
2 Months ago
Cognite - Senior Data Scientist

Cognite

Bengaluru, Karnataka, India (Hybrid)
8 Months ago
Electronic Arts - Senior Product Manager - EA Sports FC

Electronic Arts

Vancouver, British Columbia, Canada (Hybrid)
1 Month ago
Luni - Head of Legal

Luni

Paris, Île-de-France, France (Hybrid)
6 Days ago
Glean - Security Engineer

Glean

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Coda - Senior Account Manager/Account Manager (Gaming Industry)

Coda

Shanghai, China (Hybrid)
1 Year ago
Epic Games - Senior Counsel, Litigation

Epic Games

Cary, North Carolina, United States (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

Jobs in San Francisco, California, United States

CyberArk - Strategic Partner Manager

CyberArk

United States (On-Site)
1 Month ago
Reveal - Deal Desk Analyst

Reveal

Chicago, Illinois, United States (On-Site)
1 Month ago
Adyen - Team Lead - Implementation

Adyen

San Francisco, California, United States (On-Site)
1 Month ago
Condé Nast - Associate Beauty Editor

Condé Nast

New York, United States (On-Site)
1 Month ago
Visual Concepts - Senior Graphics Engineer - NBA 2K

Visual Concepts

Novato, California, United States (Remote)
1 Week ago
Apple - Generative AI, Machine Learning Engineer

Apple

Cupertino, California, United States (On-Site)
1 Month ago
Infosys - Lead Java Fullstack Developer

Infosys

San Leandro, California, United States (On-Site)
1 Month ago
Apple - NLP Solutions Software Engineer

Apple

Cupertino, California, United States (On-Site)
1 Month ago
Microsoft - Member of Technical Staff - Full Stack Software Engineer

Microsoft

Redmond, Washington, United States (Hybrid)
2 Months ago
Kavalirio - Project Integrator

Kavalirio

Tysons, Virginia, United States (On-Site)
5 Days ago

Get notifed when new similar jobs are uploaded

Software Development & Engineering Jobs

Nintendo - Intern – Installers Software Engineer (NTD)

Nintendo

Redmond, Washington, United States (On-Site)
7 Months ago
Lytx,  Inc  - Software Engineer II

Lytx, Inc

San Diego, California, United States (On-Site)
1 Month ago
Nagarro - Associate Engineer

Nagarro

New York, New York, United States (On-Site)
1 Year ago
IBKR External - Software Engineer

IBKR External

Hyderabad, Telangana, India (Hybrid)
1 Month ago
Novomatic - Engineering Lead

Novomatic

Zabierzów, Lesser Poland Voivodeship, Poland (Remote)
1 Week ago
NVIDIA - Senior Physical Design Full Chip STA Engineer

NVIDIA

Tel Aviv-Yafo, Tel Aviv District, Israel (On-Site)
4 Months ago
AECOM - Highway Engineering Technical Lead

AECOM

Philadelphia, Pennsylvania, United States (Hybrid)
2 Weeks ago
Rolls-Royce - FPGA Engineer

Rolls-Royce

Glasgow, Scotland, United Kingdom (Hybrid)
1 Month ago
tonies studios - (Senior) Software Engineer

tonies studios

Düsseldorf, North Rhine-Westphalia, Germany (Remote)
4 Months ago
broadcom - ASIC Implementation Engineer

broadcom

San Jose, California, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

About The Company

New York, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Scale AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug