Software Engineer - Reliability

6 Months ago • 5 Years + • Software Development & Engineering • $150,000 PA - $300,000 PA

Job Summary

Job Description

Luma AI is seeking a Software Engineer - Reliability to join their Infrastructure and Research teams, focusing on managing their extensive GPU clusters. These clusters, comprising thousands of H100 GPUs across multiple providers, are crucial for training, data processing, and inference. The role requires a highly skilled SRE to ensure cluster health and develop monitoring and management tools for optimal utilization. The ideal candidate will be adept at diving deep into performance and maintenance issues within these clusters. Responsibilities include collaborating with researchers and engineers on infrastructure requirements, scaling and maintaining GPU clusters with cloud providers, designing scalable infrastructure solutions, implementing monitoring systems for proactive issue identification, building fault-tolerant designs, automating repetitive tasks, participating in on-call rotations, and developing SLOs and SLIs.
Must have:
  • 5+ years of reliability engineering experience
  • Proficiency in GPU cloud infrastructure
  • Proficiency in programming/scripting languages
  • Experience with containerization and orchestration (Kubernetes)
  • Knowledge of IaC tools (Terraform, CloudFormation)
  • Excellent problem-solving skills
  • Strong communication and collaboration skills
  • Experience with observability tools (DataDog, Prometheus, Grafana)
Good to have:
  • Experience as an SRE in AI/ML space

Job Details

The SRE role at Luma AI sits with the Infrastructure and Research teams and is responsible for our GPU clusters. Luma runs on '000s of H100 GPUs across multiple providers and clusters for Training, Data Processing and Inference. We need a highly skilled SRE to ensure those clusters are healthy and to build the monitoring and management tools we need to make full use of them. Successful candidates will want to get extremely in the weeds solving performance and maintenance problems in our clusters.

Responsibilities

  • Collaborate with researchers and engineers to specify the availability, performance, correctness, and efficiency requirements of the current and future versions of our GPU infrastructure.

  • Work with multiple GPU cloud providers to scale up, scale down, maintain and monitor our 000's GPUs in many clusters.

  • Design and implement solutions to ensure the scalability of our infrastructure to meet rapidly increasing demands.

  • Implement and manage monitoring systems to proactively identify issues and anomalies in our production environment.

  • Implement fault-tolerant and resilient design patterns to minimize service disruptions.

  • Build and maintain automation tools to streamline repetitive tasks and improve system reliability.

  • Participate in an on-call rotation to respond to critical incidents and ensure 24/7 system availability alongside other infrastructure developers.

  • Develop and maintain service level objectives (SLOs) and service level indicators (SLIs) to measure and ensure system reliability.

Experience

  • Proven work experience 5+ yrs as an reliability engineer, production engineer, infrastructure software engineer or a similar role in a fast-paced, rapidly scaling company.

  • Strong proficiency in GPU cloud infrastructure, including the underlying concepts of scheduling, scaling, cloud storage, networking and security.

  • Proficiency in programming/scripting languages.

  • Experience with containerization technologies and container orchestration platforms like Kubernetes or equivalent.

  • Knowledge of IaC tools such as Terraform or CloudFormation or equivalent.

  • Excellent problem-solving and troubleshooting skills.

  • Strong communication and collaboration skills.

  • Experience with observability tools; examples include DataDog, Prometheus, Grafana, Splunk and ELK stack or similar.

  • Knowledge of security best practices in cloud environments.

  • Good to have experience as an SRE within the AI/ML space is strongly preferred.

Your application is reviewed by real people.

Similar Jobs

Raid Base - Senior Artist, VFX

Raid Base

Los Angeles, California, United States (Remote)
12 Months ago
Dynamis Inc - Data Architect

Dynamis Inc

Huntsville, Alabama, United States (On-Site)
1 Month ago
Integrant - Senior Power Apps Engineer

Integrant

Cairo, Cairo Governorate, Egypt (Hybrid)
1 Month ago
PwC - Senior Associate in Transfer Pricing

PwC

Belgrade, Serbia (Hybrid)
4 Weeks ago
Steel City Interactive - Senior Game Designer – Multiplayer

Steel City Interactive

Sheffield, England, United Kingdom (Hybrid)
3 Months ago
Motorola solutions - Telecom Field Service Engineer

Motorola solutions

Antofagasta, Antofagasta, Chile (On-Site)
1 Year ago
Ansys - Spring 2026 Lumerical Software Development Intern - Photonics (MS/PHD)

Ansys

Vancouver, British Columbia, Canada (On-Site)
3 Months ago
Logrhytm - Senior Engineering Manager

Logrhytm

California, United States (On-Site)
1 Month ago
zoox - Endpoint Engineer

zoox

Foster City, California, United States (On-Site)
1 Month ago
Qualcomm - Engineer Staff - Gstreamer Plugin Development

Qualcomm

Hyderabad, Telangana, India (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Shield AI - V-BAT Air Vehicle Operator

Shield AI

Dallas, Texas, United States (On-Site)
3 Weeks ago
Nintendo - CONTRACT - Software Engineer - Multimedia Technology (NTD)

Nintendo

Redmond, Washington, United States (On-Site)
8 Months ago
Sumo logic - Talent Pipeline - Technical Support & Customer Success

Sumo logic

India (Remote)
1 Month ago
T systems - Python Developer

T systems

Pune, Maharashtra, India (On-Site)
1 Month ago
bytedance - Cloud Native Infrastructure Engineer

bytedance

Singapore (On-Site)
2 Months ago
IBKR External - Network Operations Engineer

IBKR External

Mumbai, Maharashtra, India (Hybrid)
2 Months ago
Square - Customer Advisor (Part-time / 15-20h, m/f/d) - Ried Weberzeile

Square

Ried Im Innkreis, Upper Austria, Austria (On-Site)
3 Weeks ago
Garena - Associate/Senior Associate, Marketing

Garena

Bengaluru, Karnataka, India (On-Site)
8 Months ago
GoTo Group - Marketing Analytics Manager

GoTo Group

Jakarta, Indonesia (On-Site)
3 Weeks ago
PlayStation Global - Character Artist (ExDev - Contract)

PlayStation Global

Los Angeles, California, United States (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Palo Alto, California, United States

Apple - Systems Design Electrical Engineer - iPhone

Apple

Cupertino, California, United States (On-Site)
2 Months ago
Universal Music Group - Manager, A&R Administration

Universal Music Group

Nashville, Tennessee, United States (On-Site)
1 Year ago
Nice - SMB Account Executive

Nice

Hoboken, New Jersey, United States (On-Site)
1 Month ago
Hasbro - Senior Accountant

Hasbro

Pawtucket, Rhode Island, United States (Hybrid)
3 Weeks ago
Marvell - Staff Engineer - Silicon Validation Engineer

Marvell

Santa Clara, California, United States (On-Site)
1 Month ago
SSC Technologies - DevOps Engineer

SSC Technologies

Kansas City, Missouri, United States (Hybrid)
2 Months ago
Open Systems Technologies - Delivery Driver

Open Systems Technologies

Locust Grove, Virginia, United States (On-Site)
1 Month ago
YouGov - VP Analytics, Data Products

YouGov

United States (Remote)
2 Months ago
The New York Times - Senior Operations Manager, International

The New York Times

New York, United States (Hybrid)
1 Month ago
Reddit - Engineering Manager, ML Feature Platform

Reddit

United States (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Software Development & Engineering Jobs

Zscaler - Senior Sales Engineer

Zscaler

New York, United States (Remote)
2 Months ago
sphere entertainment - Senior Firmware Engineer

sphere entertainment

Bozeman, Montana, United States (On-Site)
1 Month ago
Zeeco, Inc. - Senior Controls Engineer

Zeeco, Inc.

Tulsa, Oklahoma, United States (On-Site)
4 Months ago
Adtran - Senior Software Engineer

Adtran

Gdynia, Pomeranian Voivodeship, Poland (Hybrid)
2 Months ago
Snyk - Senior Software Engineer

Snyk

Lisbon, Lisbon, Portugal (Hybrid)
1 Month ago
bytedance - SOC Prototype FW Engineer- Pico - San Jose

bytedance

San Jose, California, United States (On-Site)
8 Months ago
Illumina - Senior Bioinformatics Software Engineer

Illumina

San Diego, California, United States (On-Site)
1 Month ago
rivos - Accelerator Design Verification Engineer

rivos

Santa Clara, California, United States (Hybrid)
4 Years ago
ReStream - Design Engineer

ReStream

Prague, Prague, Czechia (Remote)
1 Year ago
GoDaddy - Principal Threat Hunting Engineer

GoDaddy

India (Remote)
3 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

Palo Alto, California, United States (Hybrid)

View All Jobs

Get notified when new jobs are added by Luma

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug