Research Engineer, ML Systems

4 Months ago • All levels • System Design

Job Summary

Job Description

As a Research Engineer on the ML Systems team at Character.AI, you'll work on cutting-edge ML training and inference systems. You'll optimize GPU cluster performance, develop technologies to fine-tune leading consumer AI models, and ensure efficient serving of LLMs handling 20K+ QPS. Responsibilities include writing efficient Triton kernels, developing prefix-aware routing algorithms, training and distilling LLMs, building a scalable distributed RLHF stack, and developing systems for efficient multimodal model training and inference. Your work directly impacts the conversational experience of millions of daily users.
Must have:
  • PhD (or equivalent) research experience
  • Write clean production system code
  • Strong understanding of modern ML techniques
  • Exceptional research or creative ML systems projects
  • Comfortable writing model development code (PyTorch)
Good to have:
  • Experience training large models using PyTorch distributed, DeepSpeed, Megatron
  • Experience with GPUs & collectives, writing kernels (Triton, CUDA, CUTLASS)
  • Experience with LLM inference systems (vLLM, FlashAttention)
  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)
  • Publications in ML and systems

Job Details

About the role and team

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI, your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.

What you'll do

The ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products. 

As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.

Example projects:

  • Write efficient Triton kernels and tune them for our specific models and hardware

  • Develop prefix-aware routing algorithms to improve serving cache hit rate

  • Train and distill LLMs to improve latency while preserving accuracy and engagements

  • Build an efficient and scalable distributed RLHF stack powering the model innovations

  • Develop systems for efficient multimodal (image gen/video gen) model training & inference

Who you are

  • "All Industry Levels": at least PhD (or equivalent) research experience

  • Write clear and clean production system code

  • Strong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)

  • Track record of exceptional research or creative ML systems projects

  • Comfortable writing model development code (PyTorch) for either training or inference

Nice to Have

  • Experience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.

  • Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).

  • Experience with LLM inference systems and literature such as vLLM and FlashAttention.

  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)

  • Publications in relevant academic journals or conferences in the field of machine learning and systems

About Character.AI

Character.AI empowers people to connect, learn and tell stories through interactive entertainment. Over 20 million people visit Character.AI every month, using our technology to supercharge their creativity and imagination. Our platform lets users engage with tens of millions of characters, enjoy unlimited conversations, and embark on infinite adventures.


In just two years, we achieved unicorn status and were honored as Google Play's AI App of the Year—a testament to our innovative technology and visionary approach.


Join us and be a part of establishing this new entertainment paradigm while shaping the future of Consumer AI!

At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.

Compensation Range: $225K - $400K

Similar Jobs

Epic Games - QA Specialist

Epic Games

Cary, North Carolina, United States (On-Site)
3 Months ago
rivos - Silicon Verification - Intern

rivos

Santa Clara, California, United States (On-Site)
9 Months ago
Riot Games - Software Engineering Manager - Content Access Platform

Riot Games

Dublin, County Dublin, Ireland (On-Site)
3 Months ago
Saviynt - IAM Technical Architect, Professional Services

Saviynt

Atlanta, Georgia, United States (Remote)
1 Year ago
Innerspace  - Python Developer

Innerspace

Innsbruck, Tyrol, Austria (On-Site)
2 Months ago
Meta - Software Engineer, Systems ML - SW/HW Co-design

Meta

New York, New York, United States (Remote)
8 Months ago
WebTech Corporation - Sr Engineer- Systems

WebTech Corporation

Bengaluru, Karnataka, India (On-Site)
1 Month ago
extreme network - Systems Engineer

extreme network

Tokyo, Japan (On-Site)
1 Month ago
HP - Systems/Software Engineer

HP

Taipei City, Taiwan (On-Site)
6 Days ago
Apple - 6G AI/ML Wireless Systems Engineer

Apple

San Diego, California, United States (On-Site)
1 Week ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Spectro Cloud - Senior Software Test Engineer

Spectro Cloud

Bengaluru, Karnataka, India (Hybrid)
1 Year ago
PwC - Senior Associate - SAP APO/SNP - RDC

PwC

Kolkata, West Bengal, India (On-Site)
10 Months ago
Capgemini - Adobe AEM

Capgemini

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Google - Data Center Facilities Technician, Electrical

Google

Vienna, Vienna, Austria (On-Site)
3 Months ago
Ansys - R&D Engineer II

Ansys

Vancouver, British Columbia, Canada (On-Site)
6 Days ago
plana technologies - Network Engineer

plana technologies

Quito, Pichincha, Ecuador (Remote)
1 Month ago
Ubisoft - Lead R&D Scientist

Ubisoft

Shanghai, Shanghai, China (On-Site)
6 Months ago
Zones - Product Analyst

Zones

Islamabad, Islamabad Capital Territory, Pakistan (On-Site)
2 Months ago
New Globe - Manager, IT Operations

New Globe

Bangui, Bangui, Central African Republic (On-Site)
2 Months ago
HCL Tech - Sr designer emi/emc design, ams verif

HCL Tech

New York, United States (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

Jobs in New York, New York, United States

Patel Greene - CEI Project Administrator

Patel Greene

Sarasota, Florida, United States (On-Site)
2 Weeks ago
Dynamis Inc - Threat Mitigation Analyst

Dynamis Inc

Washington, District Of Columbia, United States (On-Site)
2 Months ago
CyberArk - AI Value Architect (PM/PO)

CyberArk

United States (Hybrid)
2 Months ago
Saronic Technologies - Foundry Software Architect

Saronic Technologies

Austin, Texas, United States (On-Site)
1 Week ago
SBM Management - Chief Technology Officer

SBM Management

Sacramento, California, United States (On-Site)
1 Month ago
eBay - Sr. TPM, Search, SEO, and Product

eBay

San Jose, California, United States (On-Site)
1 Week ago
Dungarvin - Mental Health Coordinator I / Lead Mental Health Specialist

Dungarvin

Andover, Minnesota, United States (On-Site)
1 Month ago
Nice - Product Manager

Nice

Sandy, Utah, United States (On-Site)
3 Weeks ago
Calix - AI Solutions Strategist – Strategic Accounts

Calix

United States (Remote)
1 Year ago
Rolls-Royce - Control System Senior Software Engineer

Rolls-Royce

Indianapolis, Indiana, United States (Hybrid)
1 Month ago

Get notifed when new similar jobs are uploaded

System Design Jobs

Scout - Senior Diagnostic Systems Engineer

Scout

Fremont, California, United States (On-Site)
1 Month ago
Qualcomm - SoC Systems Performance Engineer, Senior

Qualcomm

San Diego, California, United States (On-Site)
1 Week ago
Apple - Hardware Systems Design Electrical Engineer - iPhone

Apple

Cupertino, California, United States (On-Site)
1 Month ago
Apple - Senior ML Engineer - Search & Recommendation Systems

Apple

Cupertino, California, United States (On-Site)
2 Months ago
Thales - Systems Engineer with EW experience

Thales

Crawley, England, United Kingdom (On-Site)
2 Months ago
Postman - Backend and Systems Engineer, Flows

Postman

New York, New York, United States (On-Site)
9 Months ago
bytedance - Staff Research Scientist/Engineer, Recommendation Systems

bytedance

San Jose, California, United States (On-Site)
1 Week ago
Scale AI - ML Research Engineer, ML Systems

Scale AI

San Francisco, California, United States (On-Site)
3 Months ago
Qualcomm - Wireless R&D Systems Engineer (Senior Staff)

Qualcomm

Colombes, Île-de-France, France (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Character is one of the world's leading personal AI platforms. Founded in 2021 by AI pioneers Noam Shazeer and Daniel De Freitas, Character is a full-stack AI company with a globally scaled direct-to-consumer platform. 

Redwood City, California, United States (Hybrid)

Redwood City, California, United States (On-Site)

Redwood City, California, United States (Hybrid)

Redwood City, California, United States (On-Site)

Redwood City, California, United States (On-Site)

New York, New York, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

Palo Alto, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Character.AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug