Software Engineer - Data Infrastructure

7 Months ago • 5 Years + • $150,000 PA - $300,000 PA

Data Analysis

Job Description

We are seeking individuals with robust Backend Data Engineering skills to construct highly efficient and resilient systems and pipelines for large-scale data processing. You will be integrated into Luma's applied research team, contributing directly to mission-critical workstreams that utilize thousands of GPUs. Responsibilities include designing, building, and automating infrastructure for data processing across multiple clusters of thousands of GPUs, collaborating with researchers to define and implement technical data requirements, and optimizing distributed loading for model training. You will also address diverse backend engineering needs across different teams and develop high-performance infrastructure for managing and utilizing large-scale datasets for model training.

Good To Have:

Experience working with visual data
Experience working closely with Machine Learning

Must Have:

5+ years of engineering experience
2+ years in petabyte-level data processing
Experience engineering large-scale data systems
Proficiency in Kubernetes, SLURM, Ray
Strong generalist Python coding skills

Perks:

Offers Equity

Add these skills to join the top 1% applicants for this job

kubernetes

python

We are looking for people with strong Backend Data Engineering capabilities to build highly efficient, resilient systems & pipelines for large-scale data processing. You’ll be part of Luma’s applied research team and work directly on mission critical work-streams utilizing thousands of GPUs.

Responsibilities

Design, build and automate infrastructure for processing data across multiple clusters of thousands of GPUs.
Work with researchers to identify and implement technical data requirements, and optimize distributed loading for model training.
Work cross-functionally for diverse backend engineering needs.
Design & build performant infrastructure to manage and leverage large-scale datasets for our model training.

Experience

Very strong generalist python coding.
Requirement of 5+ years of engineering, including 2+ years of work experience in petabyte-level data processing.
Experience engineering large-scale systems that process and serve petabytes of data.
Deep understanding of Kubernetes, SLURM, Ray and other cluster orchestration systems.
Experience working with visual data.Experience working closely with ML is a strong plus .

Set alerts for more jobs like Software Engineer - Data Infrastructure

Set alerts for new jobs by Luma

Set alerts for new Data Analysis jobs in United States

Set alerts for new jobs in United States

Set alerts for Data Analysis (Remote) jobs