Research Scientist / Engineer – Realtime Interactive

The Realtime Interactive team at Luma is building a new paradigm for generated content, enabling users to interact with model outputs during generation. This role involves collaborating with Fundamental Research, Data, and Performance Optimization teams to train fast, efficient realtime models as World Simulators. Responsibilities include fine-tuning large-scale generative models for realtime generation, designing novel algorithms for autoregressive visual generation and temporal consistency, developing interactive applications with low latency, and processing data to create advanced interactive controls for World Modeling.

Must Have

• Work on top of pretrained multimodal generative models to fine-tune and optimize them for realtime generation
• Design novel algorithms and techniques to solve problems with autoregressive visual generation, long-range temporal consistency, and long-term memory
• Develop interactive applications with tight latency constraints
• Process data to develop advanced interactive capabilities and controls for World Modeling, such as controlling character and camera movement, audio, and more
• Experience with fine-tuning large-scale generative models
• Proficiency in PyTorch and distributed training frameworks

Good to Have

• Strong background in methods for optimizing model inference (distillation, quantization, sparsity, compression, etc.)
• Experience in gathering, processing, and annotating datasets

About the Role

At Luma, the Realtime Interactive team is responsible for building an entirely new paradigm for generated content, allowing users to interact with a model’s outputs as they are being generated. The team works closely with the Fundamental Research, Data, and Performance Optimization teams to train fast and efficient realtime models as World Simulators and beyond.

Responsibilities

• Work on top of pretrained multimodal generative models to fine-tune and optimize them for realtime generation
• Design novel algorithms and techniques to solve problems with autoregressive visual generation, long-range temporal consistency, and long-term memory
• Develop interactive applications with tight latency constraints
• Process data to develop advanced interactive capabilities and controls for World Modeling, such as controlling character and camera movement, audio, and more

Experience

• Experience with fine-tuning large-scale generative models
• Proficiency in PyTorch and distributed training frameworks
• (Preferred) Strong background in methods for optimizing model inference (distillation, quantization, sparsity, compression, etc.)
• (Preferred) Experience in gathering, processing, and annotating datasets