Research Engineer, ML Systems (All Industry Levels)

5 Months ago • All levels • Artificial Intelligence

Job Summary

Job Description

As a Research Engineer on the ML Systems team at Character.AI, you'll optimize the performance and efficiency of GPU clusters, develop new technologies to fine-tune leading consumer AI models, and serve 20K+ QPS in production with LLMs. Responsibilities include writing efficient Triton kernels, developing prefix-aware routing algorithms, training and distilling LLMs, building a scalable distributed RLHF stack, and developing systems for efficient multimodal model training and inference. Your work will directly impact millions of users daily, shaping the future of conversational AI.
Must have:
  • PhD or equivalent research experience
  • Write clean production system code
  • Strong understanding of modern ML techniques
  • Exceptional research or creative ML systems projects
  • Comfortable with PyTorch for model development
Good to have:
  • Experience training large models using PyTorch distributed, DeepSpeed, Megatron
  • GPU & collectives experience (training, serving, debugging) and kernel writing (Triton, CUDA, CUTLASS)
  • Experience with LLM inference systems (vLLM, FlashAttention)
  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)
  • Publications in relevant academic journals or conferences

Job Details

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI, your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.

About the role

The ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products. 

As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.

Example projects:

  • Write efficient Triton kernels and tune them for our specific models and hardware

  • Develop prefix-aware routing algorithms to improve serving cache hit rate

  • Train and distill LLMs to improve latency while preserving accuracy and engagements

  • Build an efficient and scalable distributed RLHF stack powering the model innovations

  • Develop systems for efficient multimodal (image gen/video gen) model training & inference

Job Requirements

  • "All Industry Levels": at least PhD (or equivalent) research experience

  • Write clear and clean production system code

  • Strong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)

  • Track record of exceptional research or creative ML systems projects

  • Comfortable writing model development code (PyTorch) for either training or inference

Nice to Have

  • Experience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.

  • Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).

  • Experience with LLM inference systems and literature such as vLLM and FlashAttention.

  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)

  • Publications in relevant academic journals or conferences in the field of machine learning and systems

About Character.AI

Founded in 2021, Character is a leading AI company offering personalized experiences through customizable AI 'Characters.' As one of the most widely used AI platforms worldwide, Character enables users to interact with AI tailored to their unique needs and preferences.

In just two years, we achieved unicorn status and were named Google Play's AI App of the Year – a testament to our groundbreaking technology and vision.

Ready to shape the future of Consumer AI? 🚀

At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.

Compensation Range: $225K - $400K

Similar Jobs

Meta - Research Scientist Intern, Language and Multimodal Research for GenAI (PhD)

Meta

Menlo Park, California, United States (On-Site)
5 Months ago
PlayStation Global - Machine Learning Engineer for Game Technology

PlayStation Global

Aliso Viejo, California, United States (On-Site)
9 Months ago
Great Learning - Data Scientist

Great Learning

Bengaluru, Karnataka, India (On-Site)
7 Months ago
Ubisoft - Senior Gameplay Programmer

Ubisoft

Shanghai, Shanghai, China (On-Site)
11 Months ago
ByteDance - Student Researcher (Doubao (Seed) - Foundation Model) - 2025 Start (PhD)

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
Interface AI - Technical Customer Success Manager

Interface AI

United States (Remote)
2 Months ago
ByteDance - Research Scientist, Vision Foundation Model

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
Meta - Research Scientist Intern, Language and Multimodal Research for MetaAI (PhD)

Meta

Bellevue, Washington, United States (On-Site)
5 Months ago
ByteDance - Research Engineer (Foundation Model) - Machine Learning Systems

ByteDance

Singapore (On-Site)
5 Months ago
Level AI - Principal Software Engineer

Level AI

Noida, Uttar Pradesh, India (Hybrid)
6 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Armada - Senior Data Engineer

Armada

Thiruvananthapuram, Kerala, India (On-Site)
6 Months ago
Google - Senior Software Engineer, Ads

Google

(On-Site)
5 Months ago
Snowed In Studios - Lead Software Developer

Snowed In Studios

Ontario, Canada (Remote)
5 Months ago
Google - Software Engineer III, Google Cloud Platforms

Google

Kirkland, Washington, United States (On-Site)
5 Months ago
Zoox - Senior/Staff Software Engineer - Machine Learning

Zoox

Boston, Massachusetts, United States (Hybrid)
6 Months ago
ByteDance - Backend Software Engineer - Privacy & Security - Singapore

ByteDance

Singapore (On-Site)
5 Months ago
NVIDIA - AI Algorithms Software Engineer (RDSS Intern)

NVIDIA

Hsinchu, Hsinchu City, Taiwan (On-Site)
3 Months ago
Netflix - Data Scientist (L5) - Product Promotion & Algorithm Performance

Netflix

Los Gatos, California, United States (On-Site)
2 Months ago
Regent Craft - Embedded Software Engineering Intern

Regent Craft

North Kingstown, Rhode Island, United States (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Jobs in New York, New York, United States

The Pokemon Company International - Assistant Controller

The Pokemon Company International

Bellevue, Washington, United States (Hybrid)
6 Months ago
Tencent - Principal Researcher: Artificial General Intelligence (Natural Language Processing)

Tencent

Bellevue, Washington, United States (On-Site)
7 Months ago
Netflix - Solutions Software Engineer (L5), Studio Tech Solutions

Netflix

Los Angeles, California, United States (On-Site)
5 Months ago
ION - Senior Technical Consultant - Endur

ION

Jersey City, New Jersey, United States (On-Site)
6 Months ago
Nintendo - Sr UX Writer

Nintendo

Redmond, Washington, United States (Hybrid)
4 Months ago
The Walt Disney Company - Manager, Software Technology

The Walt Disney Company

Glendale, California, United States (On-Site)
2 Months ago
Trek - Assembler

Trek

Columbus, Ohio, United States (On-Site)
2 Months ago
Nintendo - Manager, Consumer Programs - Nintendo Stores

Nintendo

Redmond, Washington, United States (Hybrid)
5 Months ago
Rackspace Technology - Enterprise Sales Executive VI

Rackspace Technology

United States (Remote)
2 Months ago
Discord - Staff Software Engineer, Machine Learning Platform

Discord

United States (Remote)
3 Months ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

CloudHire - ML Engineer

CloudHire

Telangana, India (Remote)
2 Months ago
Social Discovery Group - Senior NLP Engineer

Social Discovery Group

Georgia (Remote)
5 Months ago
Krafton  - Deep Learning Engineer - LLM Game Agent

Krafton

Seoul, South Korea (On-Site)
2 Months ago
Zoox - Senior Software Engineer - High Performance Computing

Zoox

Foster City, California, United States (Hybrid)
6 Months ago
Crunchyroll - Senior Machine Learning Engineer, Recommendations

Crunchyroll

Culver City, California, United States (On-Site)
5 Months ago
Interface AI - Technical Delivery Manager

Interface AI

Hyderabad, Telangana, India (Remote)
2 Months ago
Canva - Senior Computer Vision Engineer - Photo AI

Canva

London, England, United Kingdom (Remote)
2 Months ago
AI Fund - Curriculum Developer

AI Fund

India (Remote)
6 Months ago
Axon - Senior Technical Program Manager, AI

Axon

Scottsdale, Arizona, United States (Remote)
2 Months ago
Zoox - Senior/Staff Motion Planning Engineer, Teleguidance

Zoox

Foster City, California, United States (Hybrid)
6 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Character is one of the world's leading personal AI platforms. Founded in 2021 by AI pioneers Noam Shazeer and Daniel De Freitas, Character is a full-stack AI company with a globally scaled direct-to-consumer platform. 

New York, New York, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

Palo Alto, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

Menlo Park, California, United States (Remote)

San Francisco, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Character.AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug