Senior Site Reliability Engineer - AI Research Clusters

3 Months ago • 5 Years + • DevOps

Job Summary

Job Description

As a Senior Site Reliability Engineer at NVIDIA, you'll be a key member of the GPU AI/HPC Infrastructure team, responsible for designing, implementing, and maintaining high-performance GPU compute clusters supporting AI research. Your responsibilities will include building large-scale automation solutions, optimizing cluster operations for reliability and efficiency, troubleshooting system failures, and collaborating with researchers on performance analysis and optimization of deep learning workflows. You will also be involved in designing, implementing, and supporting operational and reliability aspects of large-scale distributed systems, focusing on performance at scale, real-time monitoring, logging, and alerting. The role requires significant experience with GPU computing, AI infrastructure, and large-scale compute environments.
Must have:
  • 5+ years experience in designing and operating large scale compute infrastructure
  • Experience in SRE for high-performance computing
  • Deep understanding of GPU computing and AI infrastructure
  • Experience with AI/HPC schedulers (e.g., Slurm)
  • Experience with cluster configuration management tools (e.g., Ansible)
  • Proficiency in Python and Bash scripting
Good to have:
  • Familiarity with NVIDIA GPUs, CUDA programming, NCCL, MLPerf benchmarking, InfiniBand
  • Experience with cloud deployment, BCM, Terraform
  • Understanding of distributed storage systems (e.g., Lustre, GPFS)
  • Multi-cloud experience

Job Details

As a member of the GPU AI/HPC Infrastructure team, you will provide leadership in the design and implementation of groundbreaking GPU compute clusters that powers all AI research across NVIDIA. We seek an expert to build and operate these clusters at high reliability, efficiency, and performance and drive foundational improvements and automation to improve researchers productivity. As a Site Reliability Engineer, you are responsible for the big picture of how our systems relate to each other, we use a breadth of tools and approaches to tackle a broad spectrum of problems. Practices such as limiting time spent on reactive operational work, blameless postmortems and proactive identification of potential outages factor into iterative improvement that is key to both product quality and interesting dynamic day-to-day work. SRE's culture of diversity, intellectual curiosity, problem solving and openness is important to our success. Our organization brings together people with a wide variety of backgrounds, experiences and perspectives. We encourage them to collaborate, think big and take risks in a blame-free environment. We promote self-direction to work on meaningful projects, while we also strive to build an environment that provides the support and mentorship needed to learn and grow.

What you'll be doing:

In this role you will be building and improving our ecosystem around GPU-accelerated computing including developing large scale automation solutions. You will also be maintaining and building deep learning AI-HPC GPU clusters at scale and supporting our researchers to run their flows on our clusters including performance analysis and optimizations of deep learning workflows. You will design, implement and support operational and reliability aspects of large scale distributed systems with focus on performance at scale, real time monitoring, logging, and alerting.

  • Design and implement state-of-the-art GPU compute clusters.

  • Optimize cluster operations for maximum reliability, efficiency, and performance.

  • Drive foundational improvements and automation to enhance researcher productivity.

  • Troubleshoot, diagnose, and root cause of system failures and isolate the components/failure scenarios while working with internal & external partners.

  • Scale systems sustainably through mechanisms like automation, and evolve systems by pushing for changes that improve reliability and velocity.

  • Practice sustainable incident response and blameless postmortems and Be part of an on-call rotation to support production systems

  • Write and review code, develop documentation and capacity plans, debug the hardest problems, live, on some of the largest and most complex systems in the world.

  • Implement remediations across software and hardware stack according to plan, while keeping a thorough procedural record and data log and Manage upgrades and automated rollbacks across all clusters.

What we need to see:

  • Bachelor’s degree in computer science, Electrical Engineering or related field or equivalent experience with a minimum 5+ years of experience designing and operating large scale compute infrastructure.

  • Proven experience in site reliability engineering for high-performance computing environments with operational experience of at least 2K GPUs cluster.

  • Deep understanding of GPU computing and AI infrastructure.

  • Passion for solving complex technical challenges and optimizing system performance.

  • Experience with AI/HPC advanced job schedulers, and ideally familiarity with schedulers such as Slurm.

  • Working knowledge of cluster configuration management tools such as BCM or Ansible and infrastructure level applications, such as Kubernetes, Terraform, MySQL, etc.

  • In depth understating of container technologies like Docker, Enroot, etc.

  • Experience programming in Python and Bash scripting.

Ways to stand out from the crowd:

  • Interest in crafting, analyzing, and fixing large-scale distributed systems.

  • Familiarity with NVIDIA GPUs, Cuda Programming, NCCL, MLPerf benchmarking, InfiniBand with IBoIP and RDMA.

  • Experience with Cloud Deployment, BCM, Terraform.

  • Understanding of fast, distributed storage systems like Lustre and GPFS for AI/HPC workloads.

  • Multi-cloud experience.

#LI-Hybrid

Similar Jobs

Garena - Site Reliability Engineer/Senior Site Reliability Engineer

Garena

Singapore (On-Site)
1 Month ago
bazzar voice  - Software Engineer (Frontend)

bazzar voice

Belfast, Northern Ireland, United Kingdom (Hybrid)
2 Months ago
NVIDIA - SOC Clock Distribution Engineer

NVIDIA

Tel Aviv-Yafo, Tel Aviv District, Israel (On-Site)
3 Months ago
Brillio - Azure DB Architect - Migration - R01531206

Brillio

Bengaluru, Karnataka, India (Hybrid)
7 Months ago
playrix  - Senior Release Support Engineer

playrix

Ireland (Remote)
7 Months ago
bytedance - Machine Learning Engineer - Inference

bytedance

San Jose, California, United States (On-Site)
2 Months ago
Remedy Entertainment Plc - Senior/Lead Build Engineer

Remedy Entertainment Plc

Helsinki, Uusimaa, Finland (Hybrid)
3 Months ago
Tencent - Version management 游戏版本管理

Tencent

Shenzhen, Guangdong Province, China (On-Site)
7 Months ago
Rackspace Technology - Azure Cloud Engineer III

Rackspace Technology

Bengaluru, Karnataka, India (Remote)
1 Month ago
ARHS - Data Manager

ARHS

Stockholm, Stockholm County, Sweden (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

bytedance - Senior/Tech Lead Network Software Development Engineer, Switch - Seattle

bytedance

Seattle, Washington, United States (On-Site)
6 Months ago
Veeam Software - Site Reliability Engineer

Veeam Software

Warsaw, Masovian Voivodeship, Poland (Remote)
2 Weeks ago
TransUnion - Software Developer

TransUnion

Heredia, Costa Rica (Remote)
2 Weeks ago
Merqube - Cloud Support Engineer (Fintech)

Merqube

(Remote)
1 Month ago
Regent craft - Senior Software Engineer (Platform, Operations, & DevOps)

Regent craft

North Kingstown, Rhode Island, United States (On-Site)
3 Months ago
NVIDIA - Senior AI-HPC Storage Engineer

NVIDIA

Santa Clara, California, United States (On-Site)
4 Months ago
Take-Two Interactive - Security Detection Engineer

Take-Two Interactive

Austin, Texas, United States (On-Site)
2 Weeks ago
Nintendo - DevOps Engineer

Nintendo

Redmond, Washington, United States (On-Site)
4 Months ago
Zazz - Cybersecurity Analyst

Zazz

(Remote)
3 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Gurugram, Haryana, India

Rohde & Schwarz - Frontend Developer

Rohde & Schwarz

Bengaluru, Karnataka, India (On-Site)
9 Months ago
conga - Technical Architect / Sr. Technical Architect

conga

India (On-Site)
2 Weeks ago
PTW - HR Executive - Operations

PTW

Hyderabad, Telangana, India (On-Site)
1 Year ago
Unity - Sales Onboarding Intern

Unity

Bengaluru, Karnataka, India (On-Site)
3 Days ago
Aeries technology - Senior Cybersecurity Analyst

Aeries technology

Hyderabad, Telangana, India (On-Site)
3 Weeks ago
Enphase Energy - Cloud Quality Engineer

Enphase Energy

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Qualcomm - Staff ASIC Design Verification Engineer

Qualcomm

Bengaluru, Karnataka, India (On-Site)
3 Weeks ago
Illuminia - Sr Staff Information Security Engineer

Illuminia

Bengaluru, Karnataka, India (On-Site)
5 Days ago
GoDaddy - Manager of SOX Compliance & Risk Management

GoDaddy

India (Remote)
1 Week ago
luxsoft - Avaloq Software Engineer (Data)

luxsoft

India (Remote)
2 Weeks ago

Get notifed when new similar jobs are uploaded

DevOps Jobs

Nintendo - CONTRACT - Sr Engineer (NTD)

Nintendo

Redmond, Washington, United States (On-Site)
6 Months ago
The Walt Disney Company - Manager, Database Reliability Engineering

The Walt Disney Company

California, United States (On-Site)
1 Month ago
Luxoft - Senior Software Support Engineer

Luxoft

(Remote)
6 Months ago
Starkflow - Oracle Integration Cloud Consultant

Starkflow

(Remote)
1 Month ago
Smilegate - Lost Ark Build Manager

Smilegate

Seongnam-si, Gyeonggi-do, South Korea (On-Site)
4 Months ago
CharacterAI - Software Engineer, Machine Learning Infrastructure

CharacterAI

New York, New York, United States (On-Site)
2 Months ago
Omnissa - Senior Member of Technical Staff (C++ Windows)

Omnissa

Chennai, Tamil Nadu, India (On-Site)
7 Months ago
Bally's Interactive - Senior Data Developer

Bally's Interactive

Toronto, Ontario, Canada (Hybrid)
1 Month ago
Tencent - Senior Database Administrator (Private Cloud)

Tencent

(On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Since its founding in 1993, NVIDIA (NASDAQ: NVDA) has been a pioneer in accelerated computing. The company’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics, ignited the era of modern AI and is fueling the creation of the metaverse. NVIDIA is now a full-stack computing company with data-center-scale offerings that are reshaping industry.

Santa Clara, California, United States (On-Site)

Massachusetts, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Texas, United States (On-Site)

Santa Clara, California, United States (Hybrid)

Santa Clara, California, United States (Hybrid)

Pune, Maharashtra, India (On-Site)

Taipei City, Taiwan (On-Site)

View All Jobs

Get notified when new jobs are added by NVIDIA

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug