Research Engineer / Scientist - Storage for LLM

3 Months ago • All levels • Research Development • $137,750 PA - $237,500 PA

Job Summary

Job Description

ByteDance seeks a Research Engineer/Scientist to design and maintain a high-performance KV cache layer for LLM inference. Responsibilities include designing a distributed KV cache system for storing and retrieving intermediate states of transformer-based LLMs, optimizing low-latency access and eviction policies for caching long-context inputs, integrating the cache with token streaming pipelines, developing cache consistency protocols, implementing memory-aware sharding and replication strategies, and monitoring system performance. The role requires expertise in distributed systems, memory management, low-latency serving, high-performance computing, and proficiency in languages like C++, Rust, Go, or CUDA. This is a crucial role impacting millions of users, with opportunities for publication and open-source contributions.
Must have:
  • PhD in relevant field
  • Transformer model internals expertise
  • Distributed systems & memory management experience
  • High-performance computing familiarity (GPUs)
  • C++, Rust, Go, or CUDA proficiency
Good to have:
  • LLM inference serving system experience
  • Memory hierarchy optimization expertise
  • GPU-to-GPU communication knowledge
  • Cache-aware scheduling experience
Perks:
  • Competitive compensation
  • Generous research resources
  • Innovation-driven culture
  • Publication opportunities
  • Open-source contributions
  • Medical, dental, vision insurance
  • 401k savings plan
  • Paid parental leave

Job Details

Responsibilities
About the Team The Infrastructure System Lab is a hybrid research and engineering group focused on building next-generation AI-native data infrastructure. Positioned at the intersection of databases, large-scale systems, and AI, the team leads innovation in areas such as vector and multi-modal databases, infrastructure optimization through machine learning, and LLM-based tooling like NL2SQL and NL2Chart. They also develop high-performance cache systems, including multi-engine key-value stores and LLM inference KV caches. The team thrives on collaboration, with researchers and engineers working closely to take ideas from paper to prototype to production. Their work supports key products used by millions and is regularly published and deployed at scale. About the Role We are seeking a systems researcher or engineer with deep expertise in large-scale distributed storage and caching infrastructure to design and maintain a high-performance KV cache layer for large language model (LLM) inference. This role focuses on improving latency, throughput, and cost-efficiency in transformer-based model serving by optimizing the reuse of attention key-value states and prompt embeddings. You’ll work on cutting-edge AI systems problems with real-world impact, alongside a world-class team. The role offers opportunities to publish, contribute to open-source, attend top conferences, and enjoy competitive compensation, generous research resources, and an innovation-driven culture. Responsibilities - Design and implement a distributed KV cache system to store and retrieve intermediate states (e.g., attention keys/values) for transformer-based LLMs across GPUs or nodes. - Optimize low-latency access and eviction policies for caching long-context LLM inputs, token streams, and reused embeddings. - Collaborate with inference and serving teams to integrate the cache with token streaming pipelines, batched decoding, and model parallelism. - Develop cache consistency and synchronization protocols for multi-tenant, multi-request environments. - Implement memory-aware sharding, eviction (e.g., windowed LRU, TTL), and replication strategies across GPUs or distributed memory backends. - Monitor system performance and iterate on caching algorithms to reduce compute costs and response time for inference workloads. - Evaluate and, where needed, extend open-source KV stores or build custom GPU-aware caching layers (e.g., CUDA, Triton, shared memory, RDMA).
Qualifications
Minimum Qualifications - PhD in Computer Science, Applied Mathematics, Electrical Engineering, or a related technical field. - Strong understanding of transformer-based model internals and how KV caching affects autoregressive decoding. - Experience with distributed systems, memory management, and low-latency serving (RPC, gRPC, CUDA-aware networking). - Familiarity with high-performance compute environments (NVIDIA GPUs, TensorRT, Triton Inference Server). - Proficiency in languages like C++, Rust, Go, or CUDA for systems-level development. Preferred Qualifications - Prior experience building inference-serving systems for LLMs (e.g., vLLM, SGLang, FasterTransformer, DeepSpeed, Hugging Face Text Generation Inference). - Experience with memory hierarchy optimization (HBM, NUMA, NVLink) and GPU-to-GPU communication (NCCL, GDR, GDS, InfiniBand). - Exposure to cache-aware scheduling, batching, and prefetching strategies in model serving.
Job Information
【For Pay Transparency】Compensation Description (Annually)

The base salary range for this position in the selected city is $137750 - $237500 annually.

Compensation may vary outside of this range depending on a number of factors, including a candidate’s qualifications, skills, competencies and experience, and location. Base pay is one part of the Total Package that is provided to compensate and recognize employees for their work, and this role may be eligible for additional discretionary bonuses/incentives, and restricted stock units.

Benefits may vary depending on the nature of employment and the country work location. Employees have day one access to medical, dental, and vision insurance, a 401(k) savings plan with company match, paid parental leave, short-term and long-term disability coverage, life insurance, wellbeing benefits, among others. Employees also receive 10 paid holidays per year, 10 paid sick days per year and 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure).

The Company reserves the right to modify or change these benefits programs at any time, with or without notice.

For Los Angeles County (unincorporated) Candidates:

Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state, and local laws including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Our company believes that criminal history may have a direct, adverse and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment:

1. Interacting and occasionally having unsupervised contact with internal/external clients and/or colleagues;

2. Appropriately handling and managing confidential information including proprietary and trade secret information and access to information technology systems; and

3. Exercising sound judgment.

About Us

Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok, Lemon8, CapCut and Pico as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content.

Why Join ByteDance

Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life - a mission we work towards every day.

As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Join us.

Diversity & Inclusion

ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.

Reasonable Accommodation

ByteDance is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or a reasonable accommodation, please reach out to us at

Similar Jobs

Marvell - Senior Principal Design Verification Engineer

Marvell

Santa Clara, California, United States (On-Site)
1 Month ago
Turbulent - Senior Gameplay Programmer

Turbulent

Montreal, Quebec, Canada (On-Site)
3 Weeks ago
Cygames - Sound Engineer / Game Engine

Cygames

Tokyo, Japan (On-Site)
2 Months ago
SimpliSafe - Staff Software Engineer (Device Cloud Services)

SimpliSafe

Boston, Massachusetts, United States (Hybrid)
3 Weeks ago
Critical mass - Senior Product Engineer - 3D

Critical mass

Calgary, Alberta, Canada (On-Site)
3 Weeks ago
Adyen - Senior Machine Learning Scientist

Adyen

Amsterdam, North Holland, Netherlands (On-Site)
3 Weeks ago
DevRev - Applied AI - Forward Deployed Architect

DevRev

United States (On-Site)
3 Weeks ago
Pinterest - Senior Staff Machine Learning Engineer

Pinterest

San Francisco, California, United States (Remote)
2 Weeks ago
bytedance - Machine Learning Engineer - MLDev

bytedance

San Jose, California, United States (On-Site)
4 Months ago
Zazz - Machine Learning Engineer

Zazz

(Remote)
5 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Marvell - Staff Validation Engineer

Marvell

New York, United States (On-Site)
1 Month ago
Playdead - Graphics Programmer

Playdead

Copenhagen, Denmark (On-Site)
11 Months ago
Ansys - Lead R&D Engineer - SCBU - PowerArtist

Ansys

Noida, Uttar Pradesh, India (On-Site)
1 Month ago
Rockstar Games - Animation R&D Programmer: Retargeting

Rockstar Games

Oakville, Ontario, Canada (On-Site)
4 Months ago
bytedance - Software Development Engineer Graduate (Intent-Based Networking) - 2025 Start (PhD)

bytedance

San Jose, California, United States (On-Site)
1 Week ago
Shield AI - Software Engineer (C++) (Vision)

Shield AI

Melbourne, Victoria, Australia (On-Site)
6 Days ago
CyberArk - Senior C++ Win Engineer

CyberArk

Israel (Hybrid)
2 Months ago
Digital extremes - Technical Designer

Digital extremes

London, Ontario, Canada (Hybrid)
4 Months ago
Qualcomm - CPU Performance & Power Analyst/Staff Engineer

Qualcomm

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Sony Interactive Entertainment - System Development Engineer (PlayStation Platform Game Content Authentication and Delivery System)

Sony Interactive Entertainment

Tokyo, Japan (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Seattle, Washington, United States

Glocomms - Design Engineer

Glocomms

San Francisco, California, United States (On-Site)
1 Month ago
bytedance - Research Scientist in Foundation Models for Science

bytedance

San Jose, California, United States (Hybrid)
3 Months ago
AeroSpike - Senior Quality Engineer

AeroSpike

Mountain View, California, United States (On-Site)
3 Months ago
Thumbtack - Senior Engineering Manager, Data

Thumbtack

United States (Remote)
3 Weeks ago
Tennr - Product Manager

Tennr

New York, New York, United States (On-Site)
2 Months ago
Axon - Senior RF Engineer II

Axon

Scottsdale, Arizona, United States (On-Site)
2 Months ago
bytedance - IT Program Manager

bytedance

San Jose, California, United States (On-Site)
1 Week ago
Sword Health - Chief Information Security Officer

Sword Health

United States (Remote)
2 Months ago
Coherent corp. - Customer Program Manager

Coherent corp.

Santa Clara, California, United States (On-Site)
1 Month ago
Ramboll3 - Principal, Air and Climate, Life Sciences Focus

Ramboll3

Ann Arbor, Michigan, United States (On-Site)
10 Months ago

Get notifed when new similar jobs are uploaded

Research Development Jobs

Crowd Strick - Senior Security Researcher

Crowd Strick

United Kingdom (Remote)
1 Week ago
Hedra - Machine Learning Engineer

Hedra

San Francisco, California, United States (On-Site)
4 Months ago
Whatnot - Machine Learning Engineer, Buyer Growth

Whatnot

San Francisco, California, United States (Remote)
3 Weeks ago
Plaid  - Experienced Machine Learning Engineer

Plaid

San Francisco, California, United States (On-Site)
7 Months ago
Illumina - Staff Software Signal Processing Engineer/Scientist

Illumina

San Diego, California, United States (On-Site)
2 Months ago
Haleon - Lead Machine Learning Engineer

Haleon

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Riot Games - Research Operation Coordinator - Global Research Operations Team (Contract)

Riot Games

Shanghai, Shanghai, China (On-Site)
3 Months ago
ISS Stoxx - Junior Research Analyst

ISS Stoxx

Makati City, Metro Manila, Philippines (Hybrid)
1 Month ago
Supabase - AI Tooling Engineer

Supabase

(Remote)
4 Weeks ago
EMA - AI Application Engineer

EMA

California, United States (On-Site)
4 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content.
View All Jobs

Get notified when new jobs are added by bytedance

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug