Research Engineer, ML Systems

2 Months ago • All levels • Artificial Intelligence

Job Summary

Job Description

As a Research Engineer on the ML Systems team at Character.AI, you'll work on cutting-edge ML training and inference systems, optimizing GPU cluster performance, and developing new technologies to fine-tune leading consumer AI models. You'll contribute to advancements in AI, impacting millions of daily users. Responsibilities include writing efficient Triton kernels, developing prefix-aware routing algorithms, training and distilling LLMs, building distributed RLHF stacks, and developing systems for efficient multimodal model training and inference. You'll collaborate across teams and shape the conversational experience of millions of users.
Must have:
  • PhD (or equivalent) research experience
  • Write clean production system code
  • Strong understanding of modern ML techniques
  • Exceptional research or creative ML systems projects
  • Comfortable writing PyTorch model code
Good to have:
  • Experience training large models using PyTorch distributed, DeepSpeed, Megatron
  • Experience with GPUs & collectives and writing kernels (Triton, CUDA, CUTLASS)
  • Experience with LLM inference systems (vLLM, FlashAttention)
  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)
  • Publications in ML and systems

Job Details

About the role and team

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI, your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.

What you'll do

The ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products. 

As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.

Example projects:

  • Write efficient Triton kernels and tune them for our specific models and hardware

  • Develop prefix-aware routing algorithms to improve serving cache hit rate

  • Train and distill LLMs to improve latency while preserving accuracy and engagements

  • Build an efficient and scalable distributed RLHF stack powering the model innovations

  • Develop systems for efficient multimodal (image gen/video gen) model training & inference

Who you are

  • "All Industry Levels": at least PhD (or equivalent) research experience

  • Write clear and clean production system code

  • Strong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)

  • Track record of exceptional research or creative ML systems projects

  • Comfortable writing model development code (PyTorch) for either training or inference

Nice to Have

  • Experience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.

  • Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).

  • Experience with LLM inference systems and literature such as vLLM and FlashAttention.

  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)

  • Publications in relevant academic journals or conferences in the field of machine learning and systems

About Character.AI

Character.AI empowers people to connect, learn and tell stories through interactive entertainment. Over 20 million people visit Character.AI every month, using our technology to supercharge their creativity and imagination. Our platform lets users engage with tens of millions of characters, enjoy unlimited conversations, and embark on infinite adventures.


In just two years, we achieved unicorn status and were honored as Google Play's AI App of the Year—a testament to our innovative technology and visionary approach.


Join us and be a part of establishing this new entertainment paradigm while shaping the future of Consumer AI!

At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.

Compensation Range: $225K - $400K

Similar Jobs

Oliver Agency - Senior SEO Content Writer

Oliver Agency

Manila, Metro Manila, Philippines (On-Site)
1 Week ago
velotio technologies  - Senior GenAI Engineer - .Net

velotio technologies

Maharashtra, India (Remote)
1 Month ago
Applike Group - Product Lead

Applike Group

Hamburg, Hamburg, Germany (Hybrid)
1 Year ago
Riot Games - Staff Software Engineer (Services) - League of Legends, Motivations

Riot Games

Dublin, County Dublin, Ireland (On-Site)
6 Months ago
Booming Games - Game Integrity Manager

Booming Games

(Remote)
3 Months ago
Airlab Inc  - Artificial Intelligence Researcher

Airlab Inc

Quebec, Canada (On-Site)
2 Months ago
NVIDIA - Deep Learning Performance Architect

NVIDIA

Hyderabad, Telangana, India (Hybrid)
3 Months ago
Meta - Software Engineer, Systems ML - SW/HW Co-design

Meta

Fremont, California, United States (Remote)
6 Months ago
NVIDIA - Principal Engineer - DL and AI Software

NVIDIA

Canada (On-Site)
3 Months ago
GoTo Group - Lead Data Scientist - KYC

GoTo Group

Singapore (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

OYELA - Fashion Model

OYELA

Gurugram, Haryana, India (On-Site)
7 Months ago
Mashgin - Senior Software Engineer, Computer Vision and Deep Learning

Mashgin

Palo Alto, California, United States (Hybrid)
7 Months ago
Qualcomm - Machine Learning Research Engineer

Qualcomm

San Diego, California, United States (On-Site)
4 Days ago
Google - Senior Software Engineer, Distributed Machine Learning

Google

Mountain View, California, United States (On-Site)
1 Month ago
AliveCor - Senior Regulatory Affairs Specialist

AliveCor

Bengaluru, Karnataka, India (Hybrid)
11 Months ago
Yahoo - Sr. Cyber Threat Investigator (Russia SME)

Yahoo

United States (Hybrid)
2 Days ago
Microsoft - Principal Researcher-Systems & Networking

Microsoft

Vancouver, British Columbia, Canada (On-Site)
1 Month ago
QuinStreet - National Accounts Sales Executive, Insurance

QuinStreet

United States (Remote)
3 Weeks ago
NVIDIA - Senior Software Engineer, RTL Optimization Tools

NVIDIA

Santa Clara, California, United States (On-Site)
1 Month ago
CloudLinux - Python Developer

CloudLinux

Tbilisi, Tbilisi, Georgia (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in New York, New York, United States

Accurate - Senior Product Marketing Manager

Accurate

United States (Remote)
3 Months ago
Fox Factory - Shipping Invoice Coordinator

Fox Factory

Coldwater, Michigan, United States (On-Site)
3 Days ago
Palo Alto Networks - Cortex Managed Business Lead

Palo Alto Networks

New Hampshire, United States (Remote)
2 Weeks ago
Axon - Revenue Accountant

Axon

Denver, Colorado, United States (Hybrid)
2 Weeks ago
Google - Open Career Opportunities at Wing

Google

Palo Alto, California, United States (On-Site)
1 Month ago
NetBrain - Network Automation Engineer

NetBrain

Atlanta, Georgia, United States (On-Site)
1 Month ago
world relief - Church & Volunteer Teams Mobilizer

world relief

Chicago, Illinois, United States (Hybrid)
3 Weeks ago
Sawhorse Productions - Data Analyst

Sawhorse Productions

California, United States (Remote)
2 Months ago
MyFitnessPal - B2B Marketing Lead

MyFitnessPal

United States (Remote)
2 Weeks ago
bytedance - Research Scientist- Foundation Model, Generative AI

bytedance

Seattle, Washington, United States (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

NVIDIA - Machine Learning Intern - 2025

NVIDIA

(On-Site)
4 Months ago
NVIDIA - Solutions Architect, Generative AI

NVIDIA

Santa Clara, California, United States (On-Site)
1 Month ago
bytedance - Research Scientist - AI Security

bytedance

San Jose, California, United States (On-Site)
1 Month ago
zoox - Director, Process & Tools, PMO

zoox

Foster City, California, United States (On-Site)
7 Months ago
Google - Applied AI Engineering Lead, Core Enterprise, Core

Google

Hyderabad, Telangana, India (On-Site)
1 Month ago
bytedance - LLM Software Engineer/Researcher (Applied Machine Learning)

bytedance

Seattle, Washington, United States (On-Site)
2 Months ago
Genies - Machine Learning Engineer, Character Animation & Motion AI

Genies

San Mateo, California, United States (On-Site)
2 Months ago
Google - Software Engineer III, AI/ML GenAI

Google

New York, New York, United States (On-Site)
1 Month ago
C3 AI - Solution Engineer

C3 AI

Bengaluru, Karnataka, India (On-Site)
7 Months ago
bytedance - Research Engineer- Foundation Model AI Platform- San Jose

bytedance

San Jose, California, United States (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Character is one of the world's leading personal AI platforms. Founded in 2021 by AI pioneers Noam Shazeer and Daniel De Freitas, Character is a full-stack AI company with a globally scaled direct-to-consumer platform. 

New York, New York, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

Palo Alto, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

San Francisco, California, United States (On-Site)

Menlo Park, California, United States (Remote)

San Francisco, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Character.AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug