Senior Software Engineer, AI Resiliency

2 Months ago • 6 Years + • Artificial Intelligence • Game AI • $184,000 PA - $287,500 PA

Job Summary

Job Description

NVIDIA seeks a Senior Software Engineer for AI Resiliency to lead development of AI software resiliency for powerful AI supercomputers. Responsibilities include implementing and optimizing software features (checkpoint-recovery, error detection, isolation, straggler/hang detection) for 100,000+ GPUs, contributing to large-scale distributed systems with C++ and Python, working on AI system error handling, and collaborating with various teams. The role involves developing and implementing robust tests, supporting production deployments, and ensuring seamless operation of AI training and inference workloads. The ideal candidate possesses strong experience in distributed systems, parallel programming, and fault tolerance.
Must have:
  • Proficiency in C++ and Python
  • 6+ years relevant experience
  • Understanding of distributed systems
  • AI framework familiarity (PyTorch, JAX/XLA)
  • Debugging and profiling tool experience
Good to have:
  • Experience with model training teams
  • CUDA, NCCL, or MPI experience
  • Checkpointing strategies knowledge
  • Large-scale AI cluster experience
  • Systems programming skills
Perks:
  • Equity
  • Benefits

Job Details

We are now looking for a Senior Software Engineer for AI Resiliency.

At NVIDIA, we are pushing the boundaries of what’s possible in AI. We are currently seeking a Senior Software Engineer to lead the development of AI software resiliency for the most powerful AI supercomputers in the world. As a member of our AI Software Resiliency team, you will play a pivotal role in defining and implementing critical resiliency features for AI supercomputers at a scale of 100,000+ GPUs. Your expertise will be crucial in driving down cluster downtime towards zero, ensuring that our AI systems remain robust and reliable at all times.

What You’ll Be Doing:

  • Develop AI Software Resiliency Features: Implement and optimize software features that improve AI system reliability at a massive scale, such as fast checkpoint-recovery, error detection, error isolation, and straggler/hang detection.

  • Hands-On Coding & Optimization: Contribute to large-scale distributed systems with high-quality, production-level C++ and Python code. Enhance performance for AI workloads running on thousands of GPUs.

  • Fault Tolerance & Debugging: Work on AI system error handling, implementing techniques to detect silent data corruption (SDC) and other failure scenarios. Assist in developing monitoring tools for proactive failure mitigation.

  • Collaborate Across Teams: Work closely with senior engineers, AI researchers, and hardware/software teams to integrate resiliency features into AI frameworks like PyTorch and JAX/XLA.

  • Testing & Automation: Develop and implement tests to ensure robustness, scalability, and efficiency of resiliency mechanisms. Contribute to CI/CD pipelines to automate validation of AI workloads.

  • Support Production Deployments: Assist in debugging and performance tuning large-scale AI workloads in cloud and HPC environments, ensuring seamless operation of AI training and inference workloads.

What We Need to See:

  • You've achieved a Bachelor’s, Master’s or PhD in Computer Science, Electrical Engineering, or a related field, or equivalent experience.

  • Proficiency in C++ and Python, with experience in writing efficient, high-performance code.

  • 6+ years of relevant experience

  • Strong understanding of distributed systems concepts, parallel programming, and fault tolerance in large-scale computing environments.

  • Familiarity with AI frameworks such as PyTorch, JAX/XLA, TensorFlow, or similar.

  • Experience with debugging and profiling tools (e.g., gdb, perf, valgrind, NVIDIA Nsight).

  • Excellent problem-solving skills and ability to work in a fast-paced, highly collaborative environment.

Ways to Stand Out From the Crowd:

  • Hands-on experience in training models or working with model training teams.

  • Hands-on experience with CUDA, NCCL, or MPI for GPU-accelerated computing, especially at extreme-scale.

  • Knowledge of checkpointing strategies, error mitigation, or fault-tolerant computing in AI training.

  • Experience working with large-scale AI clusters, HPC environments, or cloud-based AI workloads.

  • Strong systems programming skills and experience with low-level performance tuning.

As part of the AI Resiliency team at NVIDIA, you’ll work alongside world-class engineers solving some of the hardest challenges in AI infrastructure. You’ll have the opportunity to contribute directly to making AI training and inference more reliable, scalable, and efficient. If you're passionate about AI, distributed systems, and high-performance computing, we want to hear from you!

The base salary range is 184,000 USD - 287,500 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Similar Jobs

Meta - Postdoctoral Researcher, Embodied AI (PhD)

Meta

Seattle, Washington, United States (On-Site)
6 Months ago
Google - Senior Software Engineer, Machine Learning (Recommendations, Rankings, and Predictions)

Google

Mountain View, California, United States (On-Site)
1 Month ago
matchgroup - Sr. Software Engineer, Machine Learning

matchgroup

Palo Alto, California, United States (Hybrid)
3 Weeks ago
PlayStation Global - Machine Learning Engineer for Game Technology

PlayStation Global

London, England, United Kingdom (On-Site)
10 Months ago
Canva - Senior Computer Vision Engineer - Photo AI

Canva

Vienna, Vienna, Austria (Remote)
4 Months ago
NVIDIA - Solution Architect - Auto

NVIDIA

Shanghai, Shanghai, China (On-Site)
3 Months ago
Google - Senior Staff Software Engineer, GPU Performance, Google Scale

Google

Kirkland, Washington, United States (On-Site)
1 Month ago
ByteDance - Researcher Graduate (Applied Machine Learning - Enterprise)

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
Google - Senior Software Engineer, Generative AI, Google Cloud AI

Google

Sunnyvale, California, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

NVIDIA - Senior System Software Engineer - Triton Inference Server

NVIDIA

California, United States (Remote)
4 Months ago
FlawlessAi - Research Engineer Internship - Audio Driven Facial Animation

FlawlessAi

Santa Monica, California, United States (Hybrid)
1 Month ago
Capgemini - Solution/Delivery Architect

Capgemini

Mumbai, Maharashtra, India (On-Site)
3 Weeks ago
ByteDance - Algorithm Engineer - Audio Understanding

ByteDance

Singapore (On-Site)
7 Months ago
NVIDIA - Senior Site Reliability Engineer - AI Research Clusters

NVIDIA

Santa Clara, California, United States (Hybrid)
4 Months ago
ByteDance - Software Engineer Graduate (Applied Machine Learning - Engine) - 2025 Start (BS/MS)

ByteDance

San Jose, California, United States (On-Site)
7 Months ago
Arrise Solutions (India)   - Data Scientist - Recommender S/m's

Arrise Solutions (India)

Hyderabad, Telangana, India (On-Site)
8 Months ago
Ubisoft - Senior ML Data Scientist

Ubisoft

Montreal, Quebec, Canada (On-Site)
2 Months ago
ByteDance - Research Scientist in Machine Learning for Science (AML - AI-for-Science) - 2024 Start (PhD)

ByteDance

Seattle, Washington, United States (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Santa Clara, California, United States

Scale AI - Product Manager, Red Team

Scale AI

San Francisco, California, United States (On-Site)
2 Weeks ago
Tencent - Senior Researcher: Artificial General Intelligence (Natural Language Processing)

Tencent

Washington, United States (On-Site)
3 Months ago
Google - Senior Hardware Engineer, Control Electronics, Quantum AI

Google

Goleta, California, United States (On-Site)
1 Month ago
Cognite - Customer Business Executive

Cognite

Houston, Texas, United States (Hybrid)
5 Months ago
Instawork - Senior Account Executive - Field Sales

Instawork

Atlanta, Georgia, United States (Hybrid)
1 Month ago
Kokotree - Product Marketing Manager

Kokotree

Wilmington, North Carolina, United States (On-Site)
6 Months ago
Zinnia - Head of Tax

Zinnia

Greenwich, Connecticut, United States (Hybrid)
1 Month ago
ByteDance - Senior Data Engineer, Global E-Commerce Governance Platform

ByteDance

San Jose, California, United States (On-Site)
3 Months ago
Nintendo - Configuration Support Specialist

Nintendo

Redmond, Washington, United States (Hybrid)
8 Months ago
The Walt Disney Company - Staff Electrical Engineer

The Walt Disney Company

Anaheim, California, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

Google - Staff Research Engineer, Applied ML

Google

London, England, United Kingdom (On-Site)
1 Month ago
NVIDIA - Senior AI-HPC Storage Engineer

NVIDIA

Westford, Massachusetts, United States (On-Site)
3 Months ago
ByteDance - Software Engineer (Applied Machine Learning - Enterprise)

ByteDance

San Jose, California, United States (On-Site)
1 Month ago
Inworld AI - Staff C++ Engineer

Inworld AI

Mountain View, California, United States (On-Site)
2 Months ago
Google - Software Engineer III, AI/ML, Core

Google

Sunnyvale, California, United States (On-Site)
1 Month ago
Tencent - IaaS Product Solution Architect

Tencent

(On-Site)
1 Month ago
Microsoft - Member of Technical Staff, AI Platform Engineer

Microsoft

Mountain View, California, United States (Hybrid)
1 Month ago
Inworld AI - AI Trainer (Contractor) - Writing & Gaming

Inworld AI

Mountain View, California, United States (Remote)
2 Months ago
ByteDance - Researcher - Large Language Models, Applied Machine Learning

ByteDance

Seattle, Washington, United States (On-Site)
2 Months ago
ION - Senior AI Engineer, Italy

ION

Pisa, Tuscany, Italy (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Since its founding in 1993, NVIDIA (NASDAQ: NVDA) has been a pioneer in accelerated computing. The company’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics, ignited the era of modern AI and is fueling the creation of the metaverse. NVIDIA is now a full-stack computing company with data-center-scale offerings that are reshaping industry.

Santa Clara, California, United States (On-Site)

Massachusetts, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Texas, United States (On-Site)

Santa Clara, California, United States (Hybrid)

Santa Clara, California, United States (Hybrid)

Pune, Maharashtra, India (On-Site)

Taipei City, Taiwan (On-Site)

View All Jobs

Get notified when new jobs are added by NVIDIA

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug