Distinguished Engineer, AI Resiliency Lead

1 Month ago • 15 Years + • Artificial Intelligence • $308,000 PA - $471,500 PA

Job Summary

Job Description

NVIDIA seeks a Distinguished Engineer to lead AI Resiliency, architecting and developing software resiliency features for training AI models on large-scale superclusters. This role involves designing modular, resilient software, innovating in areas like in-memory checkpointing and anomaly detection to ensure near-zero downtime. The successful candidate will collaborate with cross-functional teams, communicate progress to senior leadership, and contribute to achieving stringent uptime requirements (downtime <1%). Responsibilities include defining scalable software architecture for resilient training across hundreds of thousands of GPUs and working with frameworks like PyTorch and JAX/XLA.
Must have:
  • Master's/Ph.D. in CS/ECE
  • 15+ years software architecture experience
  • Deep understanding of AI-optimized systems
  • 5+ years HPC/AI software development
  • Strong collaboration & communication skills
Good to have:
  • Experience with large-scale AI supercomputing applications
  • 5+ years with PyTorch and JAX/XLA
  • System architecture design expertise (CPU, GPU, memory, storage, networking)
  • HPC software development best practices implementation
Perks:
  • Equity
  • Benefits

Job Details

We are seeking a Distinguished Engineer to lead AI Resiliency at NVIDIA!

Join NVIDIA and help push the boundaries of AI. In this role, you will architect, design, and develop world-class software resiliency features for training ground breaking AI models on the largest AI superclusters in the world. Leading a team of cross-functional experts, you will drive and shape our end-to-end AI software stack, ensuring seamless training of frontier models on industry-leading frameworks like PyTorch and JAX/XLA, with near-zero downtime. Your optimizations will span from algorithmic innovations to robust software architecture, with a significant impact on NVIDIA’s most critical customers. This highly visible role demands exceptional technical expertise and leadership across organizations, with direct exposure to NVIDIA's senior leadership.

What You'll Be Doing:

  • Define a scalable software architecture to enable single-job resilient training on hundreds of thousands of GPUs with minimal downtime.

  • Design and deliver modular, resilient software features to support large-scale AI training for our top customers.

  • Innovate and evolve resilient architecture designs to achieve stringent uptime requirements (downtime < 1%), through solutions like in-memory check-pointing, in-process restart, and anomaly/SDC detection.

  • Collaborate closely with internal partners, spearheading successful project execution and communicating regular progress updates to senior leadership.

What We Need to See:

  • A Master’s or Ph.D. in Computer Science, Electrical or Computer Engineering from a top-tier university, or equivalent experience.

  • 15+ years of experience in software architecture or related fields, with a deep understanding of AI-optimized systems.

  • Excellent and proven ability to collaborate and communicate effectively across multiple engineering teams.

  • At least 5 years of hands-on experience in software development on high-complexity projects involving HPC or AI.

Ways to Stand Out from the Crowd:

  • Proven experience with large-scale AI supercomputing applications, particularly in the training phase.

  • 5+ years of experience with using and contributing to modern AI frameworks like PyTorch and JAX/XLA, specifically for large-scale training workloads.

  • A strong passion for designing system architectures tailored for AI, covering CPU, GPU, memory, storage, and networking.

  • Hands-on involvement in the entire lifecycle—from design to deployment—of large-scale High-Performance Computing (HPC) systems.

  • Experience in implementing HPC software development best practices in large-scale systems.

NVIDIA continues to expand its presence in the Datacenter space, and our team plays a pivotal role in enhancing the value of our rapidly growing datacenter deployments. We also drive a data-driven approach to hardware design and system software development. You will collaborate with a wide array of teams across NVIDIA, including deep learning research, CUDA kernel and framework development, and silicon architecture. NVIDIA is widely recognized as one of the technology industry’s most desirable employers, with some of the most dedicated and innovative minds working with us. If you’re creative, driven, and autonomous, we want to hear from you!

The base salary range is 308,000 USD - 471,500 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Similar Jobs

NVIDIA - Senior Site Reliability Engineer - Infrastructure

NVIDIA

Westford, Massachusetts, United States (On-Site)
1 Month ago
The Walt Disney Company - Principal Software Engineer, AI Architecture - Ad Platforms

The Walt Disney Company

Santa Monica, California, United States (On-Site)
1 Week ago
N-iX - AI Engineer

N-iX

Poland (Remote)
1 Week ago
NVIDIA - Solution Architect - Auto

NVIDIA

Beijing, Beijing, China (On-Site)
2 Months ago
Epic Games - Machine Learning Engineer

Epic Games

London, England, United Kingdom (On-Site)
1 Week ago
Interface AI - Engineering Manager

Interface AI

Hyderabad, Telangana, India (Remote)
4 Months ago
Zoox - Director of Perception

Zoox

Foster City, California, United States (Hybrid)
5 Months ago
NVIDIA - AI Digital Human Development Intern - 2025

NVIDIA

Hong Kong, Hong Kong (On-Site)
2 Months ago
Stylumia - Senior Machine Learning Engineer - Time Series & Computer Vision

Stylumia

Bengaluru, Karnataka, India (Hybrid)
7 Months ago
Granicus - Data Scientist 4

Granicus

Bengaluru, Karnataka, India (Hybrid)
5 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

NVIDIA - Senior Software Engineer - Distributed Inference

NVIDIA

California, United States (Remote)
2 Weeks ago
Google - Senior Software Engineer, Core Machine Learning, Google Cloud

Google

New York, New York, United States (On-Site)
4 Months ago
NVIDIA - Senior Deep Learning Performance Architect

NVIDIA

Canada (On-Site)
3 Weeks ago
Social Discovery Group - Senior NLP Engineer

Social Discovery Group

Poland (Remote)
5 Months ago
NVIDIA - Enterprise Partner Marketing Manager

NVIDIA

Taipei City, Taiwan (On-Site)
1 Month ago
The Walt Disney Company - Sr Machine Learning Engineer

The Walt Disney Company

San Francisco, California, United States (On-Site)
4 Months ago
Altagram Group - Data Science Internship/Workstudent

Altagram Group

Germany (On-Site)
6 Days ago
NVIDIA - Offensive Hardware Security Researcher

NVIDIA

Santa Clara, California, United States (On-Site)
2 Months ago
Tencent - Research Intern (NLP)

Tencent

Palo Alto, California, United States (On-Site)
1 Month ago
ByteDance - Software Engineer, ML System Scheduling

ByteDance

Seattle, Washington, United States (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Redmond, Washington, United States

Lionsgate Games - Manager, Research

Lionsgate Games

Santa Monica, California, United States (On-Site)
2 Months ago
Google - Senior Software Engineer, Machine Learning, YouTube

Google

Mountain View, California, United States (On-Site)
4 Months ago
The Walt Disney Company - Sr Software Engineer (Roku/BrightScript/SceneGraph)

The Walt Disney Company

Glendale, California, United States (On-Site)
4 Months ago
Onward Search - Graphic Designer

Onward Search

Somerville, New Jersey, United States (Remote)
1 Week ago
Canva - Head of AI Research

Canva

San Francisco, California, United States (Remote)
2 Months ago
Maxis Studios - Creative Director

Maxis Studios

Redwood City, California, United States (Hybrid)
4 Months ago
GameChanger  - Senior Product Manager, Video Platform

GameChanger

United States (Remote)
1 Week ago
Evolution - In-Studio Online Casino Dealer- Overnight ONLY 11pm-7am

Evolution

Philadelphia, Pennsylvania, United States (On-Site)
10 Months ago
NVIDIA - Senior Platform Software Engineer, PCIe

NVIDIA

Santa Clara, California, United States (On-Site)
2 Months ago
Genies - Full Stack Web Developer Intern

Genies

Los Angeles, California, United States (Hybrid)
6 Days ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

Meta - Software Engineer, Systems ML - SW/HW Co-design

Meta

Austin, Texas, United States (On-Site)
4 Months ago
Genies - Machine Learning Engineer, Character Animation & Motion AI

Genies

Los Angeles, California, United States (On-Site)
1 Month ago
Genies - Backend Engineer Intern (LLM)

Genies

San Mateo, California, United States (Hybrid)
6 Days ago
Meta - Software Engineer, Machine Learning

Meta

Fremont, California, United States (Remote)
4 Months ago
Google - Senior Software Engineer, Machine Learning, Google Cloud Compute

Google

Sunnyvale, California, United States (On-Site)
4 Months ago
Social Discovery Group - Senior NLP Engineer

Social Discovery Group

(Remote)
2 Months ago
Codeway - AI Product Engineer

Codeway

İstanbul, Türkiye (On-Site)
1 Month ago
ByteDance - Senior Software Engineer - Generative AI

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
Magic Media - Senior Automation Engineer

Magic Media

İstanbul, İstanbul, Türkiye (Remote)
6 Days ago
NVIDIA - Senior Field Application Engineer

NVIDIA

California, United States (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

About The Company

Since its founding in 1993, NVIDIA (NASDAQ: NVDA) has been a pioneer in accelerated computing. The company’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics, ignited the era of modern AI and is fueling the creation of the metaverse. NVIDIA is now a full-stack computing company with data-center-scale offerings that are reshaping industry.


Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (Hybrid)

Santa Clara, California, United States (Hybrid)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Ra'anana, Center District, Israel (On-Site)

Ra'anana, Center District, Israel (On-Site)

Yokne'am Illit, North District, Israel (On-Site)

Tel Aviv-Yafo, Tel Aviv District, Israel (On-Site)

View All Jobs

Get notified when new jobs are added by NVIDIA

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug