Senior Software Engineer, AI Resiliency

1 Month ago • 6 Years + • Artificial Intelligence • Game AI • $184,000 PA - $287,500 PA

Job Summary

Job Description

NVIDIA seeks a Senior Software Engineer for AI Resiliency to lead development of AI software resiliency for powerful AI supercomputers. Responsibilities include implementing and optimizing software features (checkpoint-recovery, error detection, isolation, straggler/hang detection) for 100,000+ GPUs, contributing to large-scale distributed systems with C++ and Python, working on AI system error handling, and collaborating with various teams. The role involves developing and implementing robust tests, supporting production deployments, and ensuring seamless operation of AI training and inference workloads. The ideal candidate possesses strong experience in distributed systems, parallel programming, and fault tolerance.
Must have:
  • Proficiency in C++ and Python
  • 6+ years relevant experience
  • Understanding of distributed systems
  • AI framework familiarity (PyTorch, JAX/XLA)
  • Debugging and profiling tool experience
Good to have:
  • Experience with model training teams
  • CUDA, NCCL, or MPI experience
  • Checkpointing strategies knowledge
  • Large-scale AI cluster experience
  • Systems programming skills
Perks:
  • Equity
  • Benefits

Job Details

We are now looking for a Senior Software Engineer for AI Resiliency.

At NVIDIA, we are pushing the boundaries of what’s possible in AI. We are currently seeking a Senior Software Engineer to lead the development of AI software resiliency for the most powerful AI supercomputers in the world. As a member of our AI Software Resiliency team, you will play a pivotal role in defining and implementing critical resiliency features for AI supercomputers at a scale of 100,000+ GPUs. Your expertise will be crucial in driving down cluster downtime towards zero, ensuring that our AI systems remain robust and reliable at all times.

What You’ll Be Doing:

  • Develop AI Software Resiliency Features: Implement and optimize software features that improve AI system reliability at a massive scale, such as fast checkpoint-recovery, error detection, error isolation, and straggler/hang detection.

  • Hands-On Coding & Optimization: Contribute to large-scale distributed systems with high-quality, production-level C++ and Python code. Enhance performance for AI workloads running on thousands of GPUs.

  • Fault Tolerance & Debugging: Work on AI system error handling, implementing techniques to detect silent data corruption (SDC) and other failure scenarios. Assist in developing monitoring tools for proactive failure mitigation.

  • Collaborate Across Teams: Work closely with senior engineers, AI researchers, and hardware/software teams to integrate resiliency features into AI frameworks like PyTorch and JAX/XLA.

  • Testing & Automation: Develop and implement tests to ensure robustness, scalability, and efficiency of resiliency mechanisms. Contribute to CI/CD pipelines to automate validation of AI workloads.

  • Support Production Deployments: Assist in debugging and performance tuning large-scale AI workloads in cloud and HPC environments, ensuring seamless operation of AI training and inference workloads.

What We Need to See:

  • You've achieved a Bachelor’s, Master’s or PhD in Computer Science, Electrical Engineering, or a related field, or equivalent experience.

  • Proficiency in C++ and Python, with experience in writing efficient, high-performance code.

  • 6+ years of relevant experience

  • Strong understanding of distributed systems concepts, parallel programming, and fault tolerance in large-scale computing environments.

  • Familiarity with AI frameworks such as PyTorch, JAX/XLA, TensorFlow, or similar.

  • Experience with debugging and profiling tools (e.g., gdb, perf, valgrind, NVIDIA Nsight).

  • Excellent problem-solving skills and ability to work in a fast-paced, highly collaborative environment.

Ways to Stand Out From the Crowd:

  • Hands-on experience in training models or working with model training teams.

  • Hands-on experience with CUDA, NCCL, or MPI for GPU-accelerated computing, especially at extreme-scale.

  • Knowledge of checkpointing strategies, error mitigation, or fault-tolerant computing in AI training.

  • Experience working with large-scale AI clusters, HPC environments, or cloud-based AI workloads.

  • Strong systems programming skills and experience with low-level performance tuning.

As part of the AI Resiliency team at NVIDIA, you’ll work alongside world-class engineers solving some of the hardest challenges in AI infrastructure. You’ll have the opportunity to contribute directly to making AI training and inference more reliable, scalable, and efficient. If you're passionate about AI, distributed systems, and high-performance computing, we want to hear from you!

The base salary range is 184,000 USD - 287,500 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Similar Jobs

Stonewall Collision & Auto Painting - Senior Data Scientist

Stonewall Collision & Auto Painting

Vijayawada, Andhra Pradesh, India (On-Site)
6 Months ago
Ubisoft - Scientifique principal en données ML _ Groupe Technologique Content Creation

Ubisoft

Montreal, Quebec, Canada (On-Site)
3 Months ago
Vigaet - Machine Learning Engineer-Internship

Vigaet

Bengaluru, Karnataka, India (On-Site)
1 Year ago
Homa games - Senior MLOps Engineer

Homa games

Île-de-France, France (On-Site)
1 Week ago
Bragg - Principal AI/ML Engineer

Bragg

Ljubljana, Ljubljana, Slovenia (Hybrid)
2 Days ago
Granicus - Data Scientist 4

Granicus

Bengaluru, Karnataka, India (Hybrid)
6 Months ago
HP - Machine Learning Engineer

HP

Palo Alto, California, United States (On-Site)
6 Months ago
Google - Software Engineer, PhD, Early Career, Campus, Machine Learning, Systems and Cloud AI, 2025 start

Google

Sunnyvale, California, United States (On-Site)
3 Months ago
Rackspace Technology - Principal MLOps Engineer

Rackspace Technology

(Remote)
3 Weeks ago
Zoox - Senior/Staff Software Engineer - 3D World Generation Pipelines

Zoox

Seattle, Washington, United States (Hybrid)
5 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

The Walt Disney Company - Lead Machine Learning Engineer

The Walt Disney Company

Seattle, Washington, United States (On-Site)
2 Weeks ago
NVIDIA - Senior System Software Engineer - AI Performance and Efficiency Tools

NVIDIA

Santa Clara, California, United States (Hybrid)
4 Weeks ago
Trendyol - Data Science Professionals - Trendyol GO

Trendyol

İzmir, İzmir, Türkiye (Hybrid)
4 Months ago
NVIDIA - Senior AI-HPC Cluster Engineer

NVIDIA

Westford, Massachusetts, United States (Hybrid)
2 Weeks ago
CloudHire - Machine Learning Engineer

CloudHire

India (Remote)
3 Weeks ago
NVIDIA - Senior Site Reliability Engineer - AI Research Clusters

NVIDIA

Austin, Texas, United States (Hybrid)
1 Month ago
ByteDance - Research Scientist, Code Generation

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
Meta - Research Scientist, Computer Vision for Generative AI (PhD)

Meta

New York, New York, United States (On-Site)
5 Months ago
NVIDIA - Research Scientist, Design Automation

NVIDIA

Austin, Texas, United States (On-Site)
2 Months ago
N-iX - Senior Data Scientist (#2665)

N-iX

Ukraine (Remote)
3 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Santa Clara, California, United States

NVIDIA - Senior Compiler Engineer - Backend

NVIDIA

Austin, Texas, United States (Remote)
1 Week ago
Lionsgate Games - Director, Financial Planning Systems

Lionsgate Games

Santa Monica, California, United States (On-Site)
1 Month ago
Discord - Staff Software Engineer - Notifications Platform

Discord

San Francisco, California, United States (Remote)
3 Months ago
On Location - Customer Service Representative

On Location

Dallas, Texas, United States (Hybrid)
2 Weeks ago
ByteDance - Senior Data Engineer, Global E-Commerce Governance Platform

ByteDance

San Jose, California, United States (On-Site)
1 Month ago
Meta - Mechanical Engineer

Meta

Austin, Texas, United States (On-Site)
5 Months ago
DraftKings - Director, Product Delivery Organization

DraftKings

Boston, Massachusetts, United States (On-Site)
1 Month ago
Mattel  Inc  - Server - Part Time

Mattel Inc

Los Angeles, California, United States (On-Site)
1 Month ago
Meta - Product Manager

Meta

Menlo Park, California, United States (Remote)
5 Months ago
ByteDance - Student Researcher (Doubao (Seed) - Foundation Model - Speech & Audio) - 2025 Start (PhD)

ByteDance

Seattle, Washington, United States (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

ByteDance - Software Engineer Intern (Doubao (Seed) - Machine Learning System) - 2025 Summer (MS)

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
The Walt Disney Company - Lead Machine Learning Engineer

The Walt Disney Company

Seattle, Washington, United States (On-Site)
2 Weeks ago
Meta - Research Scientist, Computer Vision for Generative AI (PhD)

Meta

New York, New York, United States (On-Site)
5 Months ago
NVIDIA - Senior Solution Engineer, Mission Control

NVIDIA

Durham, North Carolina, United States (On-Site)
2 Weeks ago
Arrise Solutions (India)   - Lead ML Engineer

Arrise Solutions (India)

Hyderabad, Telangana, India (On-Site)
6 Months ago
Electronic Arts - Associate, Corporate Strategy

Electronic Arts

Redwood City, California, United States (Hybrid)
2 Months ago
ByteDance - AI Security Researcher - Security Flow

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
DEVOTEAM - Data Driven | MLOps Engineer

DEVOTEAM

Lisbon, Lisbon, Portugal (Remote)
5 Months ago
Keywords Studios - Research Associate - Fresher

Keywords Studios

Bengaluru, Karnataka, India (On-Site)
5 Days ago
Keywords Studios - Technical Research Associate - AI

Keywords Studios

(Remote)
3 Weeks ago

Get notifed when new similar jobs are uploaded

About The Company

Since its founding in 1993, NVIDIA (NASDAQ: NVDA) has been a pioneer in accelerated computing. The company’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics, ignited the era of modern AI and is fueling the creation of the metaverse. NVIDIA is now a full-stack computing company with data-center-scale offerings that are reshaping industry.


Santa Clara, California, United States (On-Site)

Texas, United States (Remote)

Santa Clara, California, United States (On-Site)

Yokne'am Illit, North District, Israel (On-Site)

United Kingdom (Remote)

Yokne'am Illit, North District, Israel (On-Site)

Bengaluru, Karnataka, India (Hybrid)

Toronto, Ontario, Canada (On-Site)

View All Jobs

Get notified when new jobs are added by NVIDIA

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug