Senior Software Engineer, AI Resiliency

2 Months ago • 6 Years + • Artificial Intelligence • Game AI • $184,000 PA - $287,500 PA

Job Summary

Job Description

NVIDIA seeks a Senior Software Engineer for AI Resiliency to lead development of AI software resiliency for its powerful AI supercomputers (100,000+ GPUs). Responsibilities include implementing and optimizing software features for enhanced reliability (checkpoint-recovery, error detection/isolation, straggler/hang detection); contributing to large-scale distributed systems using C++ and Python; working on AI system error handling and silent data corruption (SDC); collaborating with engineers and researchers; developing and implementing robust tests; and supporting production deployments in cloud and HPC environments. This role demands expertise in distributed systems, parallel programming, fault tolerance, and AI frameworks like PyTorch and JAX/XLA.
Must have:
  • Proficiency in C++ and Python
  • 6+ years of relevant experience
  • Understanding of distributed systems
  • Familiarity with AI frameworks (PyTorch, JAX/XLA)
  • Experience with debugging and profiling tools
Good to have:
  • Experience training models
  • Experience with CUDA, NCCL, or MPI
  • Knowledge of checkpointing strategies
  • Experience with large-scale AI clusters
  • Strong systems programming skills
Perks:
  • Equity
  • Benefits

Job Details

We are now looking for a Senior Software Engineer for AI Resiliency.

At NVIDIA, we are pushing the boundaries of what’s possible in AI. We are currently seeking a Senior Software Engineer to lead the development of AI software resiliency for the most powerful AI supercomputers in the world. As a member of our AI Software Resiliency team, you will play a pivotal role in defining and implementing critical resiliency features for AI supercomputers at a scale of 100,000+ GPUs. Your expertise will be crucial in driving down cluster downtime towards zero, ensuring that our AI systems remain robust and reliable at all times.

What You’ll Be Doing:

  • Develop AI Software Resiliency Features: Implement and optimize software features that improve AI system reliability at a massive scale, such as fast checkpoint-recovery, error detection, error isolation, and straggler/hang detection.

  • Hands-On Coding & Optimization: Contribute to large-scale distributed systems with high-quality, production-level C++ and Python code. Enhance performance for AI workloads running on thousands of GPUs.

  • Fault Tolerance & Debugging: Work on AI system error handling, implementing techniques to detect silent data corruption (SDC) and other failure scenarios. Assist in developing monitoring tools for proactive failure mitigation.

  • Collaborate Across Teams: Work closely with senior engineers, AI researchers, and hardware/software teams to integrate resiliency features into AI frameworks like PyTorch and JAX/XLA.

  • Testing & Automation: Develop and implement tests to ensure robustness, scalability, and efficiency of resiliency mechanisms. Contribute to CI/CD pipelines to automate validation of AI workloads.

  • Support Production Deployments: Assist in debugging and performance tuning large-scale AI workloads in cloud and HPC environments, ensuring seamless operation of AI training and inference workloads.

What We Need to See:

  • You've achieved a Bachelor’s, Master’s or PhD in Computer Science, Electrical Engineering, or a related field, or equivalent experience.

  • Proficiency in C++ and Python, with experience in writing efficient, high-performance code.

  • 6+ years of relevant experience

  • Strong understanding of distributed systems concepts, parallel programming, and fault tolerance in large-scale computing environments.

  • Familiarity with AI frameworks such as PyTorch, JAX/XLA, TensorFlow, or similar.

  • Experience with debugging and profiling tools (e.g., gdb, perf, valgrind, NVIDIA Nsight).

  • Excellent problem-solving skills and ability to work in a fast-paced, highly collaborative environment.

Ways to Stand Out From the Crowd:

  • Hands-on experience in training models or working with model training teams.

  • Hands-on experience with CUDA, NCCL, or MPI for GPU-accelerated computing, especially at extreme-scale.

  • Knowledge of checkpointing strategies, error mitigation, or fault-tolerant computing in AI training.

  • Experience working with large-scale AI clusters, HPC environments, or cloud-based AI workloads.

  • Strong systems programming skills and experience with low-level performance tuning.

As part of the AI Resiliency team at NVIDIA, you’ll work alongside world-class engineers solving some of the hardest challenges in AI infrastructure. You’ll have the opportunity to contribute directly to making AI training and inference more reliable, scalable, and efficient. If you're passionate about AI, distributed systems, and high-performance computing, we want to hear from you!

The base salary range is 184,000 USD - 287,500 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Similar Jobs

Google - Software Engineer, Computer Vision and Machine Learning

Google

New Taipei, New Taipei City, Taiwan (On-Site)
1 Month ago
Adobe - Senior Machine Learning Engineer - II

Adobe

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Canva - Machine Learning Engineer Lead - User Voice

Canva

Auckland, Auckland, New Zealand (Remote)
1 Month ago
Unity - Senior Machine Learning/MLOps Developer

Unity

Montreal, Quebec, Canada (On-Site)
8 Months ago
ByteDance - Algorithm Engineer - Enterprise Solution RD

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
Google - Senior Staff Software Engineer, AI/ML GenAI, Google Ads

Google

Mountain View, California, United States (On-Site)
1 Month ago
Inworld AI - AI Trainer (Contractor) - Writing & Gaming

Inworld AI

Vancouver, British Columbia, Canada (Remote)
2 Months ago
Egnyte - Principal Machine Learning Engineer - AI

Egnyte

India (Remote)
2 Months ago
SmileGate - AI Developer ([LOST ARK Mobile])

SmileGate

Seongnam-si, Gyeonggi-do, South Korea (On-Site)
4 Months ago
Google - Senior Software Engineer, Distributed Machine Learning

Google

Mountain View, California, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Google - Customer Engineer, Applied and Generative AI, Google Cloud

Google

Singapore, Singapore (On-Site)
1 Month ago
IMC - Machine Learning Engineer

IMC

Amsterdam, North Holland, Netherlands (On-Site)
1 Month ago
Google - Technical Program Manager III, AI/ML, Cloud AI Systems

Google

Austin, Texas, United States (On-Site)
1 Month ago
ByteDance - DevOps Engineer - Applied Machine Learning Engine (Singapore)

ByteDance

Singapore (On-Site)
6 Months ago
NVIDIA - Senior Math Libraries Engineers - Python APIs

NVIDIA

Remote, Oregon, United States (Remote)
3 Months ago
ByteDance - Software Engineer in Large Model System Graduate (Machine Learning Sys-US) - 2024 Start (BS/MS)

ByteDance

San Jose, California, United States (On-Site)
7 Months ago
PwC - Senior AI Developer - Roma [DIG]

PwC

Rome, Lazio, Italy (On-Site)
8 Months ago
Xsolla - Machine Learning Engineer

Xsolla

Montreal, Quebec, Canada (Remote)
1 Month ago
Luxoft - Senior ML Engineer

Luxoft

Poland, Ohio, United States (Remote)
5 Months ago
NVIDIA - Senior Site Reliability Engineer - AI Research Clusters

NVIDIA

Westford, Massachusetts, United States (Hybrid)
3 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Redmond, Washington, United States

Nordson Corporation - Maintenance Technician

Nordson Corporation

Allen, Texas, United States (On-Site)
3 Weeks ago
Microsoft - Research Intern - High Performance and Scalable Database Systems

Microsoft

Mountain View, California, United States (On-Site)
1 Month ago
Apple - Software Engineer - ASE Data Services

Apple

Seattle, Washington, United States (On-Site)
3 Weeks ago
ByteDance - US Payroll Tax Specialist

ByteDance

San Jose, California, United States (On-Site)
3 Months ago
Snorkel AI - Payments Analyst

Snorkel AI

Redwood City, California, United States (On-Site)
2 Weeks ago
Unisys - Associate Analyst OCM

Unisys

Salt Lake City, Utah, United States (On-Site)
4 Weeks ago
NVIDIA - Senior Physical Design Engineer

NVIDIA

Santa Clara, California, United States (On-Site)
3 Months ago
DataVisor - Sr. Technical Account Manager

DataVisor

Mountain View, California, United States (Hybrid)
1 Month ago
Powerintegration - MarCom Coordinator

Powerintegration

San Jose, California, United States (On-Site)
4 Months ago
Jane Street - Enterprise Applications Engineer

Jane Street

New York, New York, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

Meta - Software Engineer, Machine Learning

Meta

Menlo Park, California, United States (On-Site)
6 Months ago
Zoox - Senior Technical Program Manager, Service Readiness

Zoox

Foster City, California, United States (On-Site)
7 Months ago
Garena - AI Image Creation Collaborator

Garena

Hanoi, Hanoi, Vietnam (On-Site)
1 Month ago
Rackspace Technology - Principal MLOps Engineer

Rackspace Technology

(Remote)
2 Months ago
ByteDance - Senior Machine Learning Engineer

ByteDance

San Jose, California, United States (On-Site)
1 Month ago
FTF Studios - FTF Senior Programmer

FTF Studios

(Remote)
1 Year ago
Egnyte - Machine Learning Engineer - AI

Egnyte

India (Remote)
2 Months ago
Meta - Research Scientist Intern, Machine Perception for Input and Interaction (PhD)

Meta

Redmond, Washington, United States (On-Site)
6 Months ago
Arrise Solutions (India)   - Senior Data Scientist (Remote)

Arrise Solutions (India)

Hyderabad, Telangana, India (Remote)
7 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Since its founding in 1993, NVIDIA (NASDAQ: NVDA) has been a pioneer in accelerated computing. The company’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics, ignited the era of modern AI and is fueling the creation of the metaverse. NVIDIA is now a full-stack computing company with data-center-scale offerings that are reshaping industry.

Santa Clara, California, United States (On-Site)

Massachusetts, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Santa Clara, California, United States (On-Site)

Texas, United States (On-Site)

Santa Clara, California, United States (Hybrid)

Santa Clara, California, United States (Hybrid)

Pune, Maharashtra, India (On-Site)

Taipei City, Taiwan (On-Site)

View All Jobs

Get notified when new jobs are added by NVIDIA

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug