Research Engineer, ML Systems

1 Month ago • All levels • Artificial Intelligence

Job Summary

Job Description

As a Research Engineer on the ML Systems team at Character.AI, you'll work on cutting-edge ML training and inference systems, optimizing GPU cluster performance, and developing new technologies. You'll fine-tune leading consumer AI models with a data flywheel, serving 20K+ QPS in production with LLMs. Responsibilities include writing efficient Triton kernels, developing prefix-aware routing algorithms, training and distilling LLMs, building a scalable distributed RLHF stack, and developing systems for efficient multimodal model training and inference. You'll collaborate across teams and improve training performance and inference runtime, shaping the conversational experience of millions of users daily.
Must have:
  • PhD (or equivalent) research experience
  • Write clean production system code
  • Strong understanding of modern ML techniques
  • Exceptional research or creative ML systems projects
  • Comfortable writing model development code (PyTorch)
Good to have:
  • Experience training large models using PyTorch distributed, DeepSpeed, Megatron
  • Experience with GPUs & collectives and writing kernels (Triton, CUDA, CUTLASS)
  • Experience with LLM inference systems (vLLM, FlashAttention)
  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)
  • Publications in relevant academic journals or conferences

Job Details

About the role and team

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI, your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.

What you'll do

The ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products. 

As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.

Example projects:

  • Write efficient Triton kernels and tune them for our specific models and hardware

  • Develop prefix-aware routing algorithms to improve serving cache hit rate

  • Train and distill LLMs to improve latency while preserving accuracy and engagements

  • Build an efficient and scalable distributed RLHF stack powering the model innovations

  • Develop systems for efficient multimodal (image gen/video gen) model training & inference

Who you are

  • "All Industry Levels": at least PhD (or equivalent) research experience

  • Write clear and clean production system code

  • Strong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)

  • Track record of exceptional research or creative ML systems projects

  • Comfortable writing model development code (PyTorch) for either training or inference

Nice to Have

  • Experience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.

  • Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).

  • Experience with LLM inference systems and literature such as vLLM and FlashAttention.

  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)

  • Publications in relevant academic journals or conferences in the field of machine learning and systems

About Character.AI

Character.AI empowers people to connect, learn and tell stories through interactive entertainment. Over 20 million people visit Character.AI every month, using our technology to supercharge their creativity and imagination. Our platform lets users engage with tens of millions of characters, enjoy unlimited conversations, and embark on infinite adventures.


In just two years, we achieved unicorn status and were honored as Google Play's AI App of the Year—a testament to our innovative technology and visionary approach.


Join us and be a part of establishing this new entertainment paradigm while shaping the future of Consumer AI!

At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.

Compensation Range: $225K - $400K

Similar Jobs

Dream Sports - Lead ML Scientist

Dream Sports

Mumbai, Maharashtra, India (On-Site)
8 Months ago
Blue Yonder - AI/ML Penetration Tester Consultant

Blue Yonder

Hyderabad, Telangana, India (On-Site)
6 Months ago
Voodoo - Head of Data

Voodoo

Paris, Île-de-France, France (Hybrid)
3 Weeks ago
Dun & Bradstreet - Digital Marketing Executive

Dun & Bradstreet

Mumbai, Maharashtra, India (On-Site)
5 Months ago
Playrix - Senior Data Analyst (Attribution)

Playrix

Portugal (Remote)
5 Months ago
Electronic Arts - Senior Manager, Generative AI Software Engineering

Electronic Arts

Austin, Texas, United States (On-Site)
1 Month ago
The Walt Disney Company - Manager, Strategic Initiatives

The Walt Disney Company

Burbank, California, United States (On-Site)
2 Months ago
Thumbtack - Staff Software Engineer,  Machine Learning Infrastructure

Thumbtack

Ontario, Canada (Remote)
5 Months ago
Google - Software Engineer III, Core Machine Learning, Google Cloud

Google

Sunnyvale, California, United States (On-Site)
5 Months ago
Keywords Studios (Player Support) - AI - Research Associate (Prompts)

Keywords Studios (Player Support)

Silesian Voivodeship, Poland (On-Site)
4 Weeks ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

ByteDance - Tech Lead Manager - Code AI

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
Head Digital Works - Data Scientist

Head Digital Works

Hyderabad, Telangana, India (On-Site)
10 Months ago
PwC - IN_Senior Associate_ Golang _Advisory Corporate_Advisory_Bangalore

PwC

Bengaluru, Karnataka, India (On-Site)
6 Months ago
Genies - Lead Applied ML Engineer, Real-time 3D Asset Optimization

Genies

Los Angeles, California, United States (On-Site)
1 Month ago
NVIDIA - Senior Digital Circuit Design Engineer

NVIDIA

Santa Clara, California, United States (On-Site)
2 Months ago
ByteDance - Software Development Engineer Graduate (Distributed NoSQL Database Systems)

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
The Walt Disney Company - Manager, Software Engineering

The Walt Disney Company

San Francisco, California, United States (On-Site)
3 Months ago
ByteDance - DevOps Engineer - Applied Machine Learning, Engine

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
Kefir Games - Senior Server Software Engineer

Kefir Games

Cyprus (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Jobs in New York, New York, United States

Oculus VR - Product Design Engineer, Reality Labs

Oculus VR

Sunnyvale, California, United States (On-Site)
9 Months ago
Singularity 6 - Senior/Staff Game Engineer

Singularity 6

United States (Hybrid)
3 Months ago
The Walt Disney Company - Senior Analyst, International Disney+ Subscriber Planning

The Walt Disney Company

Santa Monica, California, United States (On-Site)
4 Months ago
Samsung Semiconductor - Senior Engineer, QA Test and Automation

Samsung Semiconductor

San Jose, California, United States (Hybrid)
1 Month ago
Entrata - Regional Vice President, Sales-Inside Sales IC role at HQ in Lehi

Entrata

Lehi, Utah, United States (On-Site)
6 Months ago
Zoox - Senior/Staff Software Engineer, ML Performance Optimization

Zoox

Foster City, California, United States (On-Site)
5 Months ago
Rivos - Senior Memory Design Engineer

Rivos

Portland, Oregon, United States (Hybrid)
5 Months ago
Polygon Labs - Lead Product Marketing Manager

Polygon Labs

United States (Remote)
1 Month ago
Nintendo - Intern - Social Media Marketing

Nintendo

Redmond, Washington, United States (On-Site)
5 Months ago
Next Level Business Services - SQL and C# developer

Next Level Business Services

Redmond, Washington, United States (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

NVIDIA - Senior Developer Relations Manager - Robotics

NVIDIA

Tokyo, Japan (On-Site)
2 Months ago
NVIDIA - Principal Engineer - DL and AI Software

NVIDIA

Santa Clara, California, United States (On-Site)
2 Months ago
Meta - Postdoctoral Researcher, Embodied AI (PhD)

Meta

Seattle, Washington, United States (On-Site)
4 Months ago
Seedify - AI Product Manager

Seedify

India (Remote)
3 Months ago
Zoox - Senior Technical Program Manager, Service Readiness

Zoox

Foster City, California, United States (On-Site)
5 Months ago
ByteDance - Research Scientist Graduate (Foundation Model - Generative AI) - 2025 Start (PhD)

ByteDance

San Jose, California, United States (On-Site)
4 Months ago
NVIDIA - Senior Solutions Architect, Generative AI - Inference

NVIDIA

California, United States (Remote)
2 Months ago
Ubisoft - Senior C++ Programmer - Machine Learning

Ubisoft

Montreal, Quebec, Canada (On-Site)
1 Month ago
Razer - Solutions Architect

Razer

Singapore (On-Site)
6 Months ago
Luma Pictures - Artificial Intelligence (AI) Technical Director (TD)

Luma Pictures

Vancouver, British Columbia, Canada (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Character is one of the world's leading personal AI platforms. Founded in 2021 by AI pioneers Noam Shazeer and Daniel De Freitas, Character is a full-stack AI company with a globally scaled direct-to-consumer platform. 

Menlo Park, California, United States (Remote)

San Francisco, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

New York, New York, United States (On-Site)

New York, New York, United States (On-Site)

New York, New York, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Character.AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug