ML Evals Engineer

About the Team

The Consumer Products Team at OpenAI is building the next generation of consumer hardware and software interfaces—the future of how people interact with AI. Our mission is to design deeply personal, multimodal experiences that make advanced AI feel natural, useful, and human. Evaluation is at the heart of this mission: reliable, insightful metrics help us understand not just whether the model is correct, but whether the interaction feels right.

About the Role

We are looking for an ML Evals Engineer to design and build the systems that measure model and product quality across the full stack—from low-level multimodal perception to the user experience. You will work at the intersection of ML, product, and hardware, creating tools that help the team translate qualitative goals into measurable signals.

This role is based in San Francisco, CA. It is fully onsite, 4 days per week, and we offer relocation assistance to new employees.

In this role, you will:

• Design and implement extensible evaluation harnesses for multimodal tasks spanning speech, vision, and text
• Build interactive visualization and analysis tools that help engineers, designers, and researchers inspect model and UX performance
• Empower product and design teams to define and extend evaluation suites aligned with real-world usage and product vision
• Automate continuous evaluation and regression tracking to ensure each model and hardware iteration improves the experience
• Collaborate with hardware, software, research, and design teams to turn qualitative goals into quantitative evaluation metrics

You might thrive in this role if you:

• Are a software engineer with hands-on experience building tools or pipelines around LLMs or multimodal models
• Are proficient in Python for backend/data workflows and familiar with TypeScript/React or similar frameworks for visualization
• Enjoy working in tight iteration loops with product and design, translating user feel into measurable metrics
• Are comfortable working across ML, product, and hardware domains where small perception or interaction improvements have outsized impact
• Are motivated by curiosity and care deeply about the quality of user experience and the systems that measure it

Preferred qualifications:

• Experience with evaluation or visualization of multimodal models (speech, vision, or sensors)
• Familiarity with hardware prototyping or embedded ML
• Background in human-in-the-loop evaluation or UX research tooling

Why this role matters

Consumer Products is OpenAI’s bridge between intelligence and embodiment—the team building the future of computing. Evaluation is how we learn what works, what feels right, and what needs to improve. As an ML Evals Engineer, you’ll create the backbone that enables us to iterate confidently on entirely new AI-powered devices and experiences.

About OpenAI

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We push the boundaries of the capabilities of AI systems and seek to safely deploy them to the world through our products. AI is an extremely powerful tool that must be created with safety and human needs at its core, and to achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the full spectrum of humanity.

We are an equal opportunity employer, and we do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, age, veteran status, disability, genetic information, or other applicable legally protected characteristic.

Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable law, including the San Francisco Fair Chance Ordinance, the Los Angeles County Fair Chance Ordinance for Employers, and the California Fair Chance Act. For unincorporated Los Angeles County workers: we reasonably believe that criminal history may have a direct, adverse and negative relationship with the following job duties, potentially resulting in the withdrawal of a conditional offer of employment: protect computer hardware entrusted to you from theft, loss or damage; return all computer hardware in your possession (including the data contained therein) upon termination of employment or end of assignment; and maintain the confidentiality of proprietary, confidential, and non-public information. In addition, job duties require access to secure and protected information technology systems and related data security obligations.

At OpenAI, we believe artificial intelligence has the potential to help people solve immense global challenges, and we want the upside of AI to be widely shared. Join us in shaping the future of technology.