Voice AI Evaluation Lead

1 Month ago • All levels • Research Development • $135,000 PA - $165,000 PA

Job Summary

Job Description

Deepgram is seeking a Voice AI Evaluation Lead to manage the performance benchmarking of their voice AI models, including speech-to-text (STT), text-to-speech (TTS), and voice agent use cases. The role involves building scalable evaluation pipelines, creating model cards, and collaborating with research, product, QA, marketing, and data labeling teams to enhance model measurement, release, and improvement. The ideal candidate will be skilled in Python, data analysis, and designing repeatable benchmarks, translating model outputs into actionable insights for product strategy and ensuring precision, fairness, and transparency in evaluations. This is a key role in defining quality and success metrics for speech AI.
Must have:
  • Experience designing, executing, and iterating on evaluation pipelines for ML models
  • Proficiency in Python and data analysis libraries
  • Ability to develop automated evaluation systems
  • Comfort working with large-scale datasets and performance metrics
  • Strong communication skills for data interpretation
  • Proven success working cross-functionally with multiple teams
Good to have:
  • Prior experience evaluating speech-related models (STT/TTS)
  • Familiarity with model documentation formats
  • Understanding of competitive benchmarking for voice AI
  • Experience contributing to internal evaluation infrastructure
  • Background in startup environments or applied research
Perks:
  • Offers Equity
  • Offers Bonus

Job Details

Company Overview

Deepgram is the leading voice AI platform for developers building speech-to-text (STT), text-to-speech (TTS) and full speech-to-speech (STS) offerings. 200,000+ developers build with Deepgram’s voice-native foundational models – accessed through APIs or as self-managed software – due to our unmatched accuracy, latency and pricing. Customers include software companies building voice products, co-sell partners working with large enterprises, and enterprises solving internal voice AI use cases. The company ended 2024 cash-flow positive with 400+ enterprise customers, 3.3x annual usage growth across the past 4 years, over 50,000 years of audio processed and over 1 trillion words transcribed. There is no organization in the world that understands voice better than Deepgram

Opportunity:

Deepgram is looking for a Voice AI Evaluation Lead to take ownership of how we benchmark and evaluate the performance of our voice AI models. This role is pivotal to the integrity and impact of our AI offerings. You’ll be building robust benchmarking pipelines, producing clear and actionable model cards, and partnering cross-functionally with research, product, QA, marketing, and data labeling to shape how our models are measured, released, and improved. If you love designing evaluations that matter, aligning metrics with product goals, and translating data into insight, this role is for you.

What You’ll Do

  • Build and maintain scalable benchmarking pipelines for model evaluations across STT, TTS, and voice agent use cases.

  • Run regular evaluations of production and pre-release models on curated, real-world datasets.

  • Partner with Research, Data, and Engineering teams to develop new evaluation methodologies and integrate them into our development cycle.

  • Design, define and refine evaluation metrics that reflect product experience, quality, and performance goals.

  • Author comprehensive model cards and internal reports outlining model strengths, weaknesses, and recommended use cases.

  • Work closely with Data Labeling Ops to source, annotate, and prepare evaluation datasets.

  • Collaborate with QA Engineers to integrate model tests into CI/CD and release workflows.

  • Support Marketing and Product with credible, data-backed comparisons to competitors.

  • Track market developments and maintain awareness of competitive benchmarks.

  • Support GTM teams with benchmarking best practices for prospects and customers.

You’ll Love This Role If You

  • Enjoy translating model outputs into human insights that guide product strategy.

  • Are motivated by precision, fairness, and transparency in evaluation.

  • Have a data-minded approach to experimentation and thrive on uncovering what’s working—and what’s not.

  • Take pride in designing clean, repeatable benchmarks that bring clarity to complex systems.

  • Get satisfaction from cross-functional collaboration, working with researchers, product teams, and engineers alike.

  • Want to shape how we define quality and success in speech AI.

  • Are excited by the idea of being a key voice in when—and how—we release new models into the world.

It’s Important To Us That You Have

  • Experience designing, executing, and iterating on evaluation pipelines for ML models

  • Proficiency in Python and data analysis libraries

  • Ability to develop automated evaluation systems—whether scripting analysis workflows or integrating with broader ML pipelines.

  • Comfort working with large-scale datasets and crafting meaningful performance metrics and visualizations.

  • Experience using LLMs or internal tooling to accelerate analysis, QA, or pipeline prototyping.

  • Strong communication skills—especially when translating raw data into structured insights, documentation, or dashboards.

  • Proven success working cross-functionally with research, engineering, QA, and product teams.

It Would Be Great if You Had

  • Prior experience evaluating speech-related models, especially STT or TTS systems.

  • Familiarity with model documentation formats (e.g., model cards, eval reports, dashboards).

  • Understanding of competitive benchmarking and landscape analysis for voice AI products.

  • Experience contributing to or owning internal evaluation infrastructure—whether integrating with existing systems or proposing new ones.

  • A background in startup environments, applied research, or AI product deployment.

Backed by prominent investors including Y Combinator, Madrona, Tiger Global, Wing VC and NVIDIA, Deepgram has raised over $85 million in total funding. If you're looking to work on cutting-edge technology and make a significant impact in the AI industry, we'd love to hear from you!

Deepgram is an equal opportunity employer. We want all voices and perspectives represented in our workforce. We are a curious bunch focused on collaboration and doing the right thing. We put our customers first, grow together and move quickly. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, gender identity or expression, age, marital status, veteran status, disability status, pregnancy, parental status, genetic information, political affiliation, or any other status protected by the laws or regulations in the locations where we operate.

We are happy to provide accommodations for applicants who need them.

Similar Jobs

legion - Platform Product Manager

legion

Bucharest, Bucharest, Romania (Hybrid)
2 Months ago
Canva - Senior Software Engineer (Python) - Data Platform

Canva

Auckland, Auckland, New Zealand (Remote)
1 Month ago
Corsair - Global Sourcing Manager

Corsair

China (On-Site)
2 Months ago
Postman - Software Engineer, IAM

Postman

Hyderabad, Telangana, India (Hybrid)
1 Month ago
Scout - Specialist, Project Calibration Engineer

Scout

Novi, Michigan, United States (On-Site)
2 Months ago
Apple - AIML - Backend Engineer, Evaluation

Apple

Santa Clara, California, United States (On-Site)
1 Month ago
Cloud Imperium Games - Senior AI Programmer

Cloud Imperium Games

Frankfurt Am Main, Hessen, Germany (On-Site)
1 Month ago
DevRev - Applied AI Engineer - Internship

DevRev

Buenos Aires, Buenos Aires, Argentina (On-Site)
1 Month ago
Apple - Neuro R&D Engineer

Apple

Cupertino, California, United States (On-Site)
2 Weeks ago
CRB workforce  - AI Python Developer

CRB workforce

Salt Lake City, Utah, United States (Remote)
3 Weeks ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

original games - LiveOps QA Specialist

original games

(Remote)
2 Months ago
Apple - Engineering Project Manager - Health Software

Apple

Sunnyvale, California, United States (On-Site)
1 Month ago
Arxanima - Look Development Artist

Arxanima

Vienna, Vienna, Austria (Remote)
2 Weeks ago
cirrus logic - Senior Manager, HR Technology - SuccessFactors

cirrus logic

Austin, Texas, United States (Hybrid)
3 Months ago
Dentsu - Cyber Security Metrics and Behavioural Analyst

Dentsu

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (Hybrid)
1 Month ago
Epic Games - Director, Data Governance

Epic Games

Cary, North Carolina, United States (On-Site)
2 Months ago
Mozilla - Director, Product & Privacy Legal

Mozilla

United States (Remote)
1 Week ago
Flexera - Senior UI Engineer

Flexera

Bengaluru, Karnataka, India (Hybrid)
9 Months ago
Apple - Software Engineer - Embedded Systems

Apple

Cupertino, California, United States (On-Site)
1 Month ago
Crunchyroll - Senior Product Manager - Access and Identity Management

Crunchyroll

San Francisco, California, United States (On-Site)
4 Months ago

Get notifed when new similar jobs are uploaded

Jobs in California, United States

Patel Greene - Civil Site Engineer

Patel Greene

Temple Terrace, Florida, United States (On-Site)
2 Weeks ago
Blinkhealth - Bilingual Vietnamese - Customer Support Specialist

Blinkhealth

Chesterfield, Missouri, United States (On-Site)
1 Month ago
Next Level Business Services - Tealium Consultant

Next Level Business Services

San Antonio, Texas, United States (On-Site)
8 Months ago
AeroSpike - Staff Software Engineer

AeroSpike

United States (Remote)
1 Month ago
Apple - Corporate FP&A Compensation and Benefits Analytics Analyst

Apple

Cupertino, California, United States (On-Site)
1 Month ago
quience - Staff Data Scientist, ML - Growth

quience

United States (Remote)
2 Months ago
Apple - Associate Creative Director, Design, Marcom

Apple

Los Angeles, California, United States (On-Site)
1 Month ago
Hasbro - Senior Compliance Specialist

Hasbro

Pawtucket, Rhode Island, United States (On-Site)
3 Weeks ago
bytedance - Research Scientist, Code Generation

bytedance

Seattle, Washington, United States (On-Site)
8 Months ago
Qualcomm - DSP Applications Software Engineer

Qualcomm

Austin, Texas, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Research Development Jobs

PayPal - Senior Staff Machine Learning Scientist

PayPal

San Jose, California, United States (Hybrid)
4 Weeks ago
Alpha Sense - Channel and Customer Research Analyst

Alpha Sense

United States (Remote)
1 Month ago
Safe security - Senior Threat Researcher

Safe security

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Salesforce - 2025 PhD Intern - AI Research, Singapore

Salesforce

Singapore, Singapore (On-Site)
9 Months ago
Passive Logic - AI Control Theory & Optimization Scientist

Passive Logic

Salt Lake City, Utah, United States (On-Site)
7 Months ago
Capgemini - Kore.AI Lead/Architect

Capgemini

Noida, Uttar Pradesh, India (On-Site)
2 Months ago
Scale AI - Engineering Manager, Generative AI

Scale AI

Argentina (On-Site)
1 Month ago
Apple - Principal Applied Research Engineer / Scientist

Apple

Cupertino, California, United States (On-Site)
1 Month ago
bytedance - Research Scientist in Foundation Model (Music) - 2025 Start (PhD)

bytedance

San Jose, California, United States (On-Site)
8 Months ago
bytedance - Research Scientist, Reinforcement Learning

bytedance

Seattle, Washington, United States (On-Site)
8 Months ago

Get notifed when new similar jobs are uploaded

About The Company

's-Hertogenbosch, North Brabant, Netherlands (On-Site)

California, United States (Remote)

California, United States (Remote)

(Remote)

California, United States (Remote)

California, United States (Remote)

's-Hertogenbosch, North Brabant, Netherlands (Remote)

View All Jobs

Get notified when new jobs are added by Deepgram

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug