Outscal Logooutscal logo

Research Engineer, ML Systems

17 Hours ago • All levels • Artificial Intelligence

Job Summary

Job Description

As a Research Engineer on the ML Systems team at Character.AI, you'll work on cutting-edge ML training and inference systems, optimizing GPU cluster performance, and developing new technologies to fine-tune leading consumer AI models. You'll contribute to advancements in AI, impacting millions of daily users. Responsibilities include writing efficient Triton kernels, developing prefix-aware routing algorithms, training and distilling LLMs, building distributed RLHF stacks, and developing systems for efficient multimodal model training and inference. You'll collaborate across teams and shape the conversational experience of millions of users.
Must have:
  • PhD (or equivalent) research experience
  • Write clean production system code
  • Strong understanding of modern ML techniques
  • Exceptional research or creative ML systems projects
  • Comfortable writing PyTorch model code
Good to have:
  • Experience training large models using PyTorch distributed, DeepSpeed, Megatron
  • Experience with GPUs & collectives and writing kernels (Triton, CUDA, CUTLASS)
  • Experience with LLM inference systems (vLLM, FlashAttention)
  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)
  • Publications in ML and systems

Job Details

About the role and team

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI, your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.

What you'll do

The ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products. 

As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.

Example projects:

  • Write efficient Triton kernels and tune them for our specific models and hardware

  • Develop prefix-aware routing algorithms to improve serving cache hit rate

  • Train and distill LLMs to improve latency while preserving accuracy and engagements

  • Build an efficient and scalable distributed RLHF stack powering the model innovations

  • Develop systems for efficient multimodal (image gen/video gen) model training & inference

Who you are

  • "All Industry Levels": at least PhD (or equivalent) research experience

  • Write clear and clean production system code

  • Strong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)

  • Track record of exceptional research or creative ML systems projects

  • Comfortable writing model development code (PyTorch) for either training or inference

Nice to Have

  • Experience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.

  • Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).

  • Experience with LLM inference systems and literature such as vLLM and FlashAttention.

  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)

  • Publications in relevant academic journals or conferences in the field of machine learning and systems

About Character.AI

Character.AI empowers people to connect, learn and tell stories through interactive entertainment. Over 20 million people visit Character.AI every month, using our technology to supercharge their creativity and imagination. Our platform lets users engage with tens of millions of characters, enjoy unlimited conversations, and embark on infinite adventures.


In just two years, we achieved unicorn status and were honored as Google Play's AI App of the Year—a testament to our innovative technology and visionary approach.


Join us and be a part of establishing this new entertainment paradigm while shaping the future of Consumer AI!

At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.

Compensation Range: $225K - $400K

Similar Jobs

Plarium - Unity Developer

Plarium

Warsaw, Masovian Voivodeship, Poland (Hybrid)
1 Month ago
The Walt Disney Company - Lead Machine Learning Engineer

The Walt Disney Company

Washington, United States (On-Site)
1 Month ago
Google - Software Engineer III, AI/ML, Google Cloud

Google

Bengaluru, Karnataka, India (On-Site)
4 Months ago
Samsung Semiconductor - Senior Principal Engineer, AI/ML Software Compiler

Samsung Semiconductor

San Jose, California, United States (Hybrid)
6 Months ago
Inkittt - Office Assistant - Working Student / Mini Job

Inkittt

Berlin, Berlin, Germany (On-Site)
16 Hours ago
Inworld AI - Forward Deployed Engineer (AI Gameplay Engineer)

Inworld AI

Mountain View, California, United States (On-Site)
6 Days ago
ByteDance - AI Model Optimization Engineer

ByteDance

San Jose, California, United States (On-Site)
2 Days ago
Coursera - Machine Learning Engineer

Coursera

India (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

ByteDance - Researcher - Large Language Models, Applied Machine Learning

ByteDance

Seattle, Washington, United States (On-Site)
2 Days ago
Hitachi - Data Scientist

Hitachi

Pune, Maharashtra, India (Remote)
5 Months ago
High Moon Studios - Senior Gameplay Engineer

High Moon Studios

California, United States (Hybrid)
2 Days ago
Twitch - Software Engineer I - iOS

Twitch

San Francisco, California, United States (On-Site)
2 Months ago
NVIDIA - System Software Engineer Intern, Autonomous Vehicles

NVIDIA

Shenzhen, Guangdong Province, China (On-Site)
1 Month ago
Regent Craft - Flight Controls Engineering Intern

Regent Craft

North Kingstown, Rhode Island, United States (On-Site)
5 Months ago
Social Discovery Group - Senior NLP Engineer

Social Discovery Group

Serbia (Remote)
4 Months ago
Google - Software Engineer III, Infrastructure, Google Cloud Security and Privacy

Google

San Francisco, California, United States (On-Site)
4 Months ago
N-iX - 2D/3D Visualization Engineer

N-iX

Ukraine (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in New York, New York, United States

X Studios,  Inc  - Production Coordinator

X Studios, Inc

Winter Park, Florida, United States (On-Site)
8 Months ago
Probably Monsters - Design Director

Probably Monsters

Washington, United States (On-Site)
3 Months ago
Universal Music - Manager, eCommerce Analytics

Universal Music

New York, New York, United States (On-Site)
4 Weeks ago
Sony Pictures Animation - FX Artist

Sony Pictures Animation

Los Angeles, California, United States (Hybrid)
1 Day ago
Sphere Entertainment Co - Project Based – After Effects Artist

Sphere Entertainment Co

Burbank, California, United States (On-Site)
2 Days ago
ByteDance - Network Engineer, Optical Long-Haul and Submarine

ByteDance

Seattle, Washington, United States (On-Site)
1 Month ago
ByteDance - Senior Machine Learning Ops Engineer, ML System

ByteDance

Seattle, Washington, United States (On-Site)
4 Months ago
PlayStation Global - QA Lead (Contract)

PlayStation Global

Los Angeles, California, United States (On-Site)
2 Months ago
Next Level Business Services - Informatica Architect

Next Level Business Services

Hightstown, New Jersey, United States (On-Site)
5 Months ago
Universal Music - Senior Manager, Privacy Operations

Universal Music

Santa Monica, California, United States (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

NVIDIA - AI and ML Infra Software Engineer, GPU Clusters

NVIDIA

Redmond, Washington, United States (On-Site)
1 Month ago
ByteDance - Student Researcher (Doubao (Seed) Foundation Model - Video Generation) - 2025 Start (PhD)

ByteDance

Seattle, Washington, United States (On-Site)
4 Months ago
Social Discovery Group - Senior/Team Lead NLP engineer

Social Discovery Group

(Remote)
3 Months ago
Google - Senior Software Engineer, AI/ML, Google Cloud AI

Google

Kirkland, Washington, United States (On-Site)
3 Months ago
Lionbridge Games - Language AI (Games) Program Manager

Lionbridge Games

Masovian Voivodeship, Poland (On-Site)
1 Month ago
ByteDance - Student Researcher (Doubao (Seed) - Foundation Model - MultiModal Generative Model)

ByteDance

San Jose, California, United States (On-Site)
2 Days ago
Playtika - LMS & Learning Ops Manager

Playtika

Israel (On-Site)
2 Days ago
Ubisoft - Senior ML Ops - Content Creation Technology Group

Ubisoft

Montreal, Quebec, Canada (On-Site)
2 Months ago
NVIDIA - Deep Learning Performance Architect

NVIDIA

Bengaluru, Karnataka, India (Hybrid)
2 Months ago
Ness Digital - AI/ML Engineer

Ness Digital

Iași, Iași County, Romania (Remote)
1 Day ago

Get notifed when new similar jobs are uploaded

About The Company

Character is one of the world's leading personal AI platforms. Founded in 2021 by AI pioneers Noam Shazeer and Daniel De Freitas, Character is a full-stack AI company with a globally scaled direct-to-consumer platform. 

New York, New York, United States (On-Site)

New York, New York, United States (On-Site)

New York, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

Menlo Park, California, United States (On-Site)

New York, New York, United States (On-Site)

New York, New York, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Character.AI

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug