Outscal Logooutscal logo

Research Engineer, ML Systems

11 Hours ago • All levels • Artificial Intelligence

Job Summary

Job Description

As a Research Engineer on the ML Systems team at Character.AI, you'll work on cutting-edge ML training and inference systems. You'll optimize GPU cluster performance, develop technologies to fine-tune leading consumer AI models, and ensure efficient serving of LLMs handling 20K+ QPS. Responsibilities include writing efficient Triton kernels, developing prefix-aware routing algorithms, training and distilling LLMs, building a scalable distributed RLHF stack, and developing systems for efficient multimodal model training and inference. Your work directly impacts the conversational experience of millions of daily users.
Must have:
  • PhD (or equivalent) research experience
  • Write clean production system code
  • Strong understanding of modern ML techniques
  • Exceptional research or creative ML systems projects
  • Comfortable writing model development code (PyTorch)
Good to have:
  • Experience training large models using PyTorch distributed, DeepSpeed, Megatron
  • Experience with GPUs & collectives, writing kernels (Triton, CUDA, CUTLASS)
  • Experience with LLM inference systems (vLLM, FlashAttention)
  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)
  • Publications in ML and systems

Job Details

About the role and team

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI, your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.

What you'll do

The ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products. 

As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.

Example projects:

  • Write efficient Triton kernels and tune them for our specific models and hardware

  • Develop prefix-aware routing algorithms to improve serving cache hit rate

  • Train and distill LLMs to improve latency while preserving accuracy and engagements

  • Build an efficient and scalable distributed RLHF stack powering the model innovations

  • Develop systems for efficient multimodal (image gen/video gen) model training & inference

Who you are

  • "All Industry Levels": at least PhD (or equivalent) research experience

  • Write clear and clean production system code

  • Strong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)

  • Track record of exceptional research or creative ML systems projects

  • Comfortable writing model development code (PyTorch) for either training or inference

Nice to Have

  • Experience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.

  • Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).

  • Experience with LLM inference systems and literature such as vLLM and FlashAttention.

  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)

  • Publications in relevant academic journals or conferences in the field of machine learning and systems

About Character.AI

Character.AI empowers people to connect, learn and tell stories through interactive entertainment. Over 20 million people visit Character.AI every month, using our technology to supercharge their creativity and imagination. Our platform lets users engage with tens of millions of characters, enjoy unlimited conversations, and embark on infinite adventures.


In just two years, we achieved unicorn status and were honored as Google Play's AI App of the Year—a testament to our innovative technology and visionary approach.


Join us and be a part of establishing this new entertainment paradigm while shaping the future of Consumer AI!

At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.

Compensation Range: $225K - $400K

Similar Jobs

Smarsh - Lead Machine Learning Engineer

Smarsh

India (Hybrid)
5 Months ago
NVIDIA - Senior Research Scientist

NVIDIA

Tel Aviv-Yafo, Tel Aviv District, Israel (On-Site)
2 Months ago
Magic Leap - Director, Calibration Software

Magic Leap

Sunnyvale, California, United States (Hybrid)
8 Months ago
NVIDIA - Solutions Architect, Agentic AI

NVIDIA

Texas, United States (Remote)
1 Month ago
NVIDIA - Deep Learning Performance Architect

NVIDIA

Bengaluru, Karnataka, India (Hybrid)
2 Months ago
NVIDIA - Solutions Architect, Agentic AI

NVIDIA

Santa Clara, California, United States (Remote)
2 Months ago
Reality Games - Machine Learning Engineer - Monopoly World

Reality Games

Kraków, Lesser Poland Voivodeship, Poland (On-Site)
3 Weeks ago
Meta - Visiting Senior Research Scientist

Meta

Paris, Île-de-France, France (On-Site)
4 Months ago
Passive Logic - AI Control Theory & Optimization Scientist

Passive Logic

Salt Lake City, Utah, United States (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Playrix - Senior Data Analyst (Marketing)

Playrix

Ireland (Remote)
1 Month ago
Larian Studios - GRAPHICS PROGRAMMER

Larian Studios

Quebec, Canada (On-Site)
3 Months ago
ByteDance - Senior Research Scientist, Data Management and Security - Infrastructure System Lab

ByteDance

San Jose, California, United States (On-Site)
20 Hours ago
Meta - Research Scientist Intern, Smart Glasses in Wearables AI (PhD)

Meta

Redmond, Washington, United States (On-Site)
4 Months ago
Henkel - Data Scientist-Intern

Henkel

Pune, Maharashtra, India (On-Site)
6 Months ago
GoMotive - Technical Lead Manager, Embedded Safety

GoMotive

United States (Remote)
1 Week ago
The Walt Disney Company - Sr Software Engineer

The Walt Disney Company

Charlotte, North Carolina, United States (On-Site)
4 Months ago
SideFX Software - Senior Compositing Software Developer/Technical Director

SideFX Software

Ontario, Canada (Hybrid)
12 Hours ago
Jam City - C++ Software Engineer

Jam City

Montevideo, Montevideo Department, Uruguay (Hybrid)
2 Months ago

Get notifed when new similar jobs are uploaded

Jobs in New York, New York, United States

100 Thieves - Backend Sales Support

100 Thieves

United States (On-Site)
9 Months ago
Next Level Business Services - SAP Concur Travel and Expense Consultant

Next Level Business Services

Pleasanton, California, United States (On-Site)
5 Months ago
NVIDIA - Senior Physical Design Methodology Engineer, Innovus Flows

NVIDIA

Austin, Texas, United States (On-Site)
2 Weeks ago
Gearbox Software - UI Artist

Gearbox Software

Frisco, Texas, United States (On-Site)
3 Months ago
Flowplay llc - Customer Service Shift Supervisors

Flowplay llc

Seattle, Washington, United States (Remote)
2 Months ago
Genies - Lead Machine Learning Engineer, 3D Gen AI & Graphics

Genies

Los Angeles, California, United States (On-Site)
3 Weeks ago
ByteDance - Research Scientist (Machine Learning for Science (AI-for-Science))

ByteDance

Seattle, Washington, United States (On-Site)
19 Hours ago
Meta - Marketing Science Partner (Financial Services)

Meta

Menlo Park, California, United States (On-Site)
4 Months ago
Snail Games - Game Scout (Business Development)

Snail Games

Beverly Hills, California, United States (On-Site)
4 Months ago
Daybreak Game Company LLC - Producer

Daybreak Game Company LLC

Renton, Washington, United States (Hybrid)
2 Months ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

NVIDIA - Deep Learning Performance Architect

NVIDIA

Hyderabad, Telangana, India (Hybrid)
1 Month ago
NVIDIA - Machine Learning Software Platform Architect

NVIDIA

Canada (On-Site)
1 Month ago
Zoox - Software Engineer - Simulation Workload Orchestration

Zoox

Foster City, California, United States (Hybrid)
5 Months ago
Wargaming - Gen AI Business Development Manager

Wargaming

Berlin, Berlin, Germany (On-Site)
1 Month ago
Meetelise - Senior Research Scientist

Meetelise

(Remote)
4 Months ago
NVIDIA - Applied Research Intern - 2025

NVIDIA

Yerevan, Yerevan, Armenia (On-Site)
1 Month ago
Reality Games - Machine Learning Engineer - Monopoly World

Reality Games

Kraków, Lesser Poland Voivodeship, Poland (On-Site)
3 Weeks ago
Zoox - Senior/Staff Motion Planning Engineer, Teleguidance

Zoox

Foster City, California, United States (Hybrid)
5 Months ago
Zoox - Software Engineer - Simulation Workload Orchestration

Zoox

Seattle, Washington, United States (Hybrid)
5 Months ago
NVIDIA - Machine Learning Software Platform Architect

NVIDIA

Santa Clara, California, United States (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Character is one of the world's leading personal AI platforms. Founded in 2021 by AI pioneers Noam Shazeer and Daniel De Freitas, Character is a full-stack AI company with a globally scaled direct-to-consumer platform. 

New York, New York, United States (On-Site)

Menlo Park, California, United States (On-Site)

New York, New York, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

New York, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Character.AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug