Software Engineer - Data Infrastructure

5 Months ago • 5 Years + • Data Analysis • $150,000 PA - $300,000 PA

Job Summary

Job Description

We are seeking individuals with robust Backend Data Engineering skills to construct highly efficient and resilient systems and pipelines for large-scale data processing. You will be integrated into Luma's applied research team, contributing directly to mission-critical workstreams that utilize thousands of GPUs. Responsibilities include designing, building, and automating infrastructure for data processing across multiple clusters of thousands of GPUs, collaborating with researchers to define and implement technical data requirements, and optimizing distributed loading for model training. You will also address diverse backend engineering needs across different teams and develop high-performance infrastructure for managing and utilizing large-scale datasets for model training.
Must have:
  • 5+ years of engineering experience
  • 2+ years in petabyte-level data processing
  • Experience engineering large-scale data systems
  • Proficiency in Kubernetes, SLURM, Ray
  • Strong generalist Python coding skills
Good to have:
  • Experience working with visual data
  • Experience working closely with Machine Learning
Perks:
  • Offers Equity

Job Details

We are looking for people with strong Backend Data Engineering capabilities to build highly efficient, resilient systems & pipelines for large-scale data processing. You’ll be part of Luma’s applied research team and work directly on mission critical work-streams utilizing thousands of GPUs.

Responsibilities

  • Design, build and automate infrastructure for processing data across multiple clusters of thousands of GPUs.

  • Work with researchers to identify and implement technical data requirements, and optimize distributed loading for model training.

  • Work cross-functionally for diverse backend engineering needs.

  • Design & build performant infrastructure to manage and leverage large-scale datasets for our model training.

Experience

  • Very strong generalist python coding.

  • Requirement of 5+ years of engineering, including 2+ years of work experience in petabyte-level data processing.

  • Experience engineering large-scale systems that process and serve petabytes of data.

  • Deep understanding of Kubernetes, SLURM, Ray and other cluster orchestration systems.

  • Experience working with visual data.Experience working closely with ML is a strong plus .

Similar Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

Similar Skill Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

Jobs in Palo Alto, California, United States

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

Data Analysis Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

About The Company

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

View All Jobs

Get notified when new jobs are added by Luma

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug