Research Engineer, ML Systems

2 Months ago • All levels • Artificial Intelligence

Job Summary

Job Description

As a Research Engineer on the ML Systems team at Character.AI, you'll work on cutting-edge ML training and inference systems. You'll optimize GPU cluster performance, develop technologies to fine-tune leading consumer AI models, and ensure efficient serving of LLMs handling 20K+ QPS. Responsibilities include writing efficient Triton kernels, developing prefix-aware routing algorithms, training and distilling LLMs, building a scalable distributed RLHF stack, and developing systems for efficient multimodal model training and inference. Your work directly impacts the conversational experience of millions of daily users.
Must have:
  • PhD (or equivalent) research experience
  • Write clean production system code
  • Strong understanding of modern ML techniques
  • Exceptional research or creative ML systems projects
  • Comfortable writing model development code (PyTorch)
Good to have:
  • Experience training large models using PyTorch distributed, DeepSpeed, Megatron
  • Experience with GPUs & collectives, writing kernels (Triton, CUDA, CUTLASS)
  • Experience with LLM inference systems (vLLM, FlashAttention)
  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)
  • Publications in ML and systems

Job Details

About the role and team

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI, your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.

What you'll do

The ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products. 

As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.

Example projects:

  • Write efficient Triton kernels and tune them for our specific models and hardware

  • Develop prefix-aware routing algorithms to improve serving cache hit rate

  • Train and distill LLMs to improve latency while preserving accuracy and engagements

  • Build an efficient and scalable distributed RLHF stack powering the model innovations

  • Develop systems for efficient multimodal (image gen/video gen) model training & inference

Who you are

  • "All Industry Levels": at least PhD (or equivalent) research experience

  • Write clear and clean production system code

  • Strong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)

  • Track record of exceptional research or creative ML systems projects

  • Comfortable writing model development code (PyTorch) for either training or inference

Nice to Have

  • Experience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.

  • Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).

  • Experience with LLM inference systems and literature such as vLLM and FlashAttention.

  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)

  • Publications in relevant academic journals or conferences in the field of machine learning and systems

About Character.AI

Character.AI empowers people to connect, learn and tell stories through interactive entertainment. Over 20 million people visit Character.AI every month, using our technology to supercharge their creativity and imagination. Our platform lets users engage with tens of millions of characters, enjoy unlimited conversations, and embark on infinite adventures.


In just two years, we achieved unicorn status and were honored as Google Play's AI App of the Year—a testament to our innovative technology and visionary approach.


Join us and be a part of establishing this new entertainment paradigm while shaping the future of Consumer AI!

At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.

Compensation Range: $225K - $400K

Similar Jobs

Visa - Staff Data Engineer

Visa

Warsaw, Masovian Voivodeship, Poland (Hybrid)
7 Months ago
London stock Exchange - Senior Research Analyst

London stock Exchange

Shanghai, China (On-Site)
1 Week ago
Treelix - Staff Software Development Engineer

Treelix

Bengaluru, Karnataka, India (On-Site)
5 Days ago
Light Speed Studios - Senior Engine Engineer

Light Speed Studios

(On-Site)
5 Months ago
Salesforce - Lead Site Reliability Engineer

Salesforce

Dublin, County Dublin, Ireland (On-Site)
3 Days ago
Genies - Machine Learning Infrastructure Engineer, 3D Model Inference & Deployment

Genies

San Mateo, California, United States (On-Site)
2 Months ago
NVIDIA - Senior Deep Learning Performance Architect

NVIDIA

Canada (On-Site)
2 Months ago
bytedance - Research Scientist Graduate (Foundation Model - Vision and Language)

bytedance

Seattle, Washington, United States (On-Site)
4 Months ago
plarium - Director of Gen-AI

plarium

Herzliya, Tel Aviv District, Israel (On-Site)
3 Months ago
Inworld AI - AI Trainer (Contractor) - Writing & Gaming

Inworld AI

Mountain View, California, United States (Remote)
2 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Google - Software Engineer III, Google Cloud Business Platforms

Google

Sunnyvale, California, United States (On-Site)
1 Month ago
Adobe - Director, Operational Intelligence

Adobe

San Jose, California, United States (On-Site)
2 Weeks ago
Google - Senior Software Engineer, Core

Google

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Google - Software Engineer, People with Disabilities

Google

State Of Minas Gerais, Brazil (On-Site)
5 Months ago
appier - Senior Software Engineer, Backend Development

appier

Taipei City, Taiwan (On-Site)
2 Weeks ago
shyft labs - Machine Learning Engineer

shyft labs

Atlanta, Georgia, United States (Hybrid)
2 Weeks ago
conga - Staff Software Engineer

conga

Pune, Maharashtra, India (On-Site)
1 Month ago
Ziff Davis - Software Developer

Ziff Davis

Canada (Remote)
4 Weeks ago
OKX - Graduate Hire 2024/25 - SRE/Security Engineer

OKX

Hong Kong (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded

Jobs in New York, New York, United States

The E.W. Scripps Company - AM Executive Producer

The E.W. Scripps Company

Salt Lake City, Utah, United States (On-Site)
1 Week ago
Diligent - Senior Field Marketing Manager

Diligent

New York, New York, United States (On-Site)
1 Month ago
bytedance - Machine Learning Engineer Intern (Search-TikTok Recommendation)

bytedance

Seattle, Washington, United States (On-Site)
2 Months ago
Everi - Principal Software Developer – Systems Engineering

Everi

Austin, Texas, United States (Hybrid)
1 Week ago
Bright Edge - Business Operations Associate II

Bright Edge

Cleveland, Ohio, United States (On-Site)
2 Months ago
Stem - Tier 2 Support Specialist

Stem

Broomfield, Colorado, United States (On-Site)
4 Days ago
Visa - Sr. Manager - Cloud Infrastructure Engineering

Visa

Ashburn, Virginia, United States (Hybrid)
2 Weeks ago
Postman - Account Development Representative

Postman

Boston, Massachusetts, United States (Hybrid)
1 Month ago
Meta - Software Engineer - Datacenter networking

Meta

Bellevue, Washington, United States (On-Site)
6 Months ago
Snap Mobile INC - Account Executive

Snap Mobile INC

Philadelphia, Pennsylvania, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

Ubisoft - Senior ML Programmer

Ubisoft

Montreal, Quebec, Canada (On-Site)
4 Months ago
Google - Software Engineer, Computer Vision and Machine Learning

Google

New Taipei, New Taipei City, Taiwan (On-Site)
1 Month ago
Quizizz - ML Engineer

Quizizz

Bengaluru, Karnataka, India (On-Site)
8 Months ago
Cricketpedia - AI Engineer

Cricketpedia

Gurugram, Haryana, India (Remote)
2 Years ago
Razer - Solutions Architect

Razer

Singapore (On-Site)
8 Months ago
bytedance - Research Scientist Graduate (Foundation Model - Vision and Language)

bytedance

Seattle, Washington, United States (On-Site)
2 Months ago
Google - Staff Image Quality Evaluation Engineer, Silicon

Google

Mountain View, California, United States (On-Site)
1 Month ago
zoox - Senior/Staff Software Engineer - Prediction Integration

zoox

Foster City, California, United States (Hybrid)
7 Months ago
NVIDIA - AI Algorithms Software Engineer (RDSS Intern)

NVIDIA

Hsinchu, Hsinchu City, Taiwan (On-Site)
4 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Character is one of the world's leading personal AI platforms. Founded in 2021 by AI pioneers Noam Shazeer and Daniel De Freitas, Character is a full-stack AI company with a globally scaled direct-to-consumer platform. 

New York, New York, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

Palo Alto, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

Menlo Park, California, United States (Remote)

San Francisco, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Character.AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug