Distinguished Engineer, AI Resiliency Lead

3 Months ago • 15 Years + • Artificial Intelligence • $308,000 PA - $471,500 PA

Job Summary

Job Description

NVIDIA seeks a Distinguished Engineer to lead AI Resiliency, architecting and developing resilient software features for training AI models on massive superclusters. This role involves designing a scalable architecture for single-job resilient training on hundreds of thousands of GPUs with minimal downtime (<1%). Responsibilities include designing modular software features, innovating resilient architectures (in-memory checkpointing, in-process restart, anomaly detection), collaborating with internal teams, and reporting to senior leadership. The ideal candidate will have 15+ years of software architecture experience, deep AI systems understanding, and expertise with frameworks like PyTorch and JAX/XLA. Experience with large-scale AI supercomputing applications and HPC is crucial.
Must have:
  • 15+ years software architecture experience
  • Deep understanding of AI-optimized systems
  • Experience with PyTorch and JAX/XLA
  • Large-scale AI supercomputing experience
  • HPC software development best practices
Good to have:
  • Experience contributing to modern AI frameworks
  • Hands-on involvement in HPC system lifecycle
  • Strong system architecture design skills
Perks:
  • Equity
  • Benefits

Job Details

We are seeking a Distinguished Engineer to lead AI Resiliency at NVIDIA!

Join NVIDIA and help push the boundaries of AI. In this role, you will architect, design, and develop world-class software resiliency features for training ground breaking AI models on the largest AI superclusters in the world. Leading a team of cross-functional experts, you will drive and shape our end-to-end AI software stack, ensuring seamless training of frontier models on industry-leading frameworks like PyTorch and JAX/XLA, with near-zero downtime. Your optimizations will span from algorithmic innovations to robust software architecture, with a significant impact on NVIDIA’s most critical customers. This highly visible role demands exceptional technical expertise and leadership across organizations, with direct exposure to NVIDIA's senior leadership.

What You'll Be Doing:

  • Define a scalable software architecture to enable single-job resilient training on hundreds of thousands of GPUs with minimal downtime.

  • Design and deliver modular, resilient software features to support large-scale AI training for our top customers.

  • Innovate and evolve resilient architecture designs to achieve stringent uptime requirements (downtime < 1%), through solutions like in-memory check-pointing, in-process restart, and anomaly/SDC detection.

  • Collaborate closely with internal partners, spearheading successful project execution and communicating regular progress updates to senior leadership.

What We Need to See:

  • A Master’s or Ph.D. in Computer Science, Electrical or Computer Engineering from a top-tier university, or equivalent experience.

  • 15+ years of experience in software architecture or related fields, with a deep understanding of AI-optimized systems.

  • Excellent and proven ability to collaborate and communicate effectively across multiple engineering teams.

  • At least 5 years of hands-on experience in software development on high-complexity projects involving HPC or AI.

Ways to Stand Out from the Crowd:

  • Proven experience with large-scale AI supercomputing applications, particularly in the training phase.

  • 5+ years of experience with using and contributing to modern AI frameworks like PyTorch and JAX/XLA, specifically for large-scale training workloads.

  • A strong passion for designing system architectures tailored for AI, covering CPU, GPU, memory, storage, and networking.

  • Hands-on involvement in the entire lifecycle—from design to deployment—of large-scale High-Performance Computing (HPC) systems.

  • Experience in implementing HPC software development best practices in large-scale systems.

NVIDIA continues to expand its presence in the Datacenter space, and our team plays a pivotal role in enhancing the value of our rapidly growing datacenter deployments. We also drive a data-driven approach to hardware design and system software development. You will collaborate with a wide array of teams across NVIDIA, including deep learning research, CUDA kernel and framework development, and silicon architecture. NVIDIA is widely recognized as one of the technology industry’s most desirable employers, with some of the most dedicated and innovative minds working with us. If you’re creative, driven, and autonomous, we want to hear from you!

The base salary range is 308,000 USD - 471,500 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Similar Jobs

NVIDIA - Senior Site Reliability Engineer - AI Research Clusters

NVIDIA

Hyderabad, Telangana, India (Hybrid)
3 Months ago
HP - Machine Learning Intern

HP

Austin, Texas, United States (On-Site)
7 Months ago
NVIDIA - Senior Physical Design Methodology Engineer, Innovus Flows

NVIDIA

Austin, Texas, United States (On-Site)
1 Month ago
ByteDance - LLM Software Engineer/Researcher (Applied Machine Learning)

ByteDance

Seattle, Washington, United States (On-Site)
1 Month ago
Samsung Semiconductor - Principal Engineer, AI/ML Software Compiler

Samsung Semiconductor

San Jose, California, United States (Hybrid)
1 Month ago
Hedra - Research Scientist

Hedra

New York, New York, United States (On-Site)
1 Month ago
NVIDIA - AI Computing Software Development Engineer, TensorRT

NVIDIA

Taipei City, Taiwan (On-Site)
3 Months ago
NVIDIA - Senior Software Engineer - Distributed Inference

NVIDIA

Texas, United States (Remote)
1 Month ago
Inkittt - Senior Machine Learning Engineer, Recommendations

Inkittt

San Francisco, California, United States (Hybrid)
3 Months ago
Meta - Research Scientist Intern, Machine Perception for Input and Interaction (PhD)

Meta

Burlingame, California, United States (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

ByteDance - Senior Machine Learning Ops Engineer, ML System

ByteDance

Seattle, Washington, United States (On-Site)
6 Months ago
NVIDIA - AI Computing Software Development Engineer, TensorRT

NVIDIA

Shanghai, Shanghai, China (On-Site)
3 Months ago
Krafton  - Deep Learning Strategy & Operations Associate

Krafton

Seoul, South Korea (On-Site)
2 Months ago
NVIDIA - Senior Physical Design Methodology Engineer, PPA Fusion Compiler

NVIDIA

Austin, Texas, United States (On-Site)
2 Months ago
NVIDIA - Design Verification Engineer - PCIE

NVIDIA

Taipei City, Taiwan (On-Site)
1 Month ago
Meta - Software Engineer, Systems ML - SW/HW Co-design

Meta

New York, New York, United States (Remote)
5 Months ago
prizepicks - Engineering Manager — Data Science Engineering

prizepicks

Atlanta, Georgia, United States (Remote)
1 Month ago
GoMotive - Software Engineer, MLOps

GoMotive

Pakistan (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in Santa Clara, California, United States

Scientific Games  - Senior Counsel – Intellectual Property and Licensing

Scientific Games

Alpharetta, Georgia, United States (On-Site)
3 Months ago
2K - Release Technician

2K

Las Vegas, Nevada, United States (On-Site)
1 Month ago
Pika - Research Engineer (Applied Research) New Grad

Pika

Palo Alto, California, United States (On-Site)
5 Months ago
Rackspace Technology - Machine Learning Architect (AWS)

Rackspace Technology

San Diego, California, United States (Remote)
1 Month ago
Samsung Semiconductor - Principal, Emulation Lead

Samsung Semiconductor

San Jose, California, United States (Hybrid)
1 Month ago
Zoox - Systems Engineer - Collision Avoidance

Zoox

Foster City, California, United States (Hybrid)
6 Months ago
ByteDance - Research Scientist, Reinforcement Learning

ByteDance

Seattle, Washington, United States (On-Site)
6 Months ago
Life church - APIs Staff Engineer

Life church

Edmond, Oklahoma, United States (On-Site)
6 Months ago
Nintendo - Sr Software Engineer - Gameplay

Nintendo

Redmond, Washington, United States (Hybrid)
1 Month ago
ByteDance - AI Product Manager

ByteDance

San Jose, California, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

Microsoft - Principal Product Manager, AI

Microsoft

Redmond, Washington, United States (Hybrid)
1 Month ago
Smarsh - Lead Machine Learning Engineer

Smarsh

India (Hybrid)
6 Months ago
AI Fund - General Manager - New Business Unit (College Admissions)

AI Fund

California, United States (Remote)
6 Months ago
Meta - Software Engineer, Machine Learning

Meta

Sunnyvale, California, United States (On-Site)
5 Months ago
Interface AI - Software Development Engineer IV - Backend

Interface AI

India (Remote)
2 Months ago
Tesla - Senior Machine Learning, AI Engineer

Tesla

Brandenburg, Germany (On-Site)
2 Months ago
Keywords Studios - Research Associate - AI

Keywords Studios

(Remote)
1 Month ago
Lionbridge Games - Games Language AI Specialist (Linguist)

Lionbridge Games

Masovian Voivodeship, Poland (On-Site)
2 Months ago
Inworld AI - Staff C++ Developer

Inworld AI

Vancouver, British Columbia, Canada (On-Site)
1 Month ago
ByteDance - Research Engineer- Foundation Model AI Platform- San Jose

ByteDance

San Jose, California, United States (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Since its founding in 1993, NVIDIA (NASDAQ: NVDA) has been a pioneer in accelerated computing. The company’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics, ignited the era of modern AI and is fueling the creation of the metaverse. NVIDIA is now a full-stack computing company with data-center-scale offerings that are reshaping industry.

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Massachusetts, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Texas, United States (On-Site)

Santa Clara, California, United States (Hybrid)

Austin, Texas, United States (Remote)

Santa Clara, California, United States (Hybrid)

View All Jobs

Get notified when new jobs are added by NVIDIA

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug