Senior GPU Cluster Software Engineer responsible for building profiling solutions for large-scale real-world applications running on GPU compute clusters.  The role involves architecting, designing, implementing, testing, deploying, and supporting large-scale distributed systems infrastructure with monitoring, logging, visualization, and alerting capabilities.  A key focus is building internal profiling tools for ML/DL applications, analyzing failures and inefficiencies to improve GPU clusters and hardware.  The engineer will also need to understand state-of-the-art improvements in ML/DL and work with application owners and research teams to enhance profiling needs for current and future features.  The work environment is agile and fast-paced, requiring collaboration in a global setting.

Explore gaming industy jobs in one of the leading Game Studios.

NVIDIA

System Software Engineer - Android Platform Security

System Software Engineering Manager

Senior Silicon Solution Engineer - Hardware

Senior System Validation Engineer

Senior System Performance and Power Engineer

Circuit Design Engineer

Senior System Integration Engineer

Senior System Software Engineer - AI Performance and Efficiency Tools

Staff Systems Software Engineer - Server

M&A and Integration Lead

<div data-automation-id="jobPostingDescription" class="css-4r17ng">As a member of the System Software team, you'll be responsible for building profiling solutions for large-scale real world applications running&nbsp; on GPU compute clusters to make them work efficiently and improve the user experience for customer as well as engineers supporting the cluster.&nbsp; Much of our software development focuses on profiling varied set of applications running on different GPU clusters, and being able to accurately measure and display the user experience on these clusters with actionable inputs for customers and engineers supporting the cluster.&nbsp; Creating a fault tolerant distributed system while minimizing data loss and limiting time spent on reactive operational work is key to product quality and dynamic day-to-day work.&nbsp; We promote self-direction to work on meaningful projects, while we also strive to build an environment that provides the support and mentorship needed to learn and grow.What you'll be doing:<ul><li>Work in an agile and fast-paced global environment to gather requirements, architect, design, implement, test, deploy, release, and support large scale distributed systems infrastructure with monitoring, logging, visualization, and alerting capabilities with promised uptime</li><li>Build internal profiling tools for real world ML/DL applications running on HPC GPU clusters for failure and efficiency analysis to help improve current and future generation of GPU clusters and associated HWs</li><li>Understand state of the art improvements in ML/DL domain, and work with various application owners and research teams to add / improve profiling needs for current and potential future supported features</li></ul>What we need to see:<ul><li>BS+ in Computer Science or related (or equivalent experience) and 5+ years of software development (in Python)</li><li>Experience with Gitlab (or another source code management) branch/release, CI/CD pipeline, etc.</li><li>Solid understanding of algorithms, data structures, and runtime/space complexity</li><li>Experience working with distributed system software architecture</li><li>Basic understanding of HPC GPU cluster, slurm</li><li>Basic understanding of Machine learning concepts and terminologies</li><li>Background with databases - SQL and NoSQL (prometheus, elasticsearch, opensearch, redis, etc.)</li><li>Experience with distributed Data Pipeline, Telemetry, Visualizations (Kibana, Grafana, etc.), Alerting (pagerduty, etc.)</li></ul>Ways to stand out from crowd:<ul><li>Experience debugging functional and performance issues in HPC GPU clusters</li><li>Background in running and instrumenting distributed LLM training on a multi gpu HPC cluster</li><li>Knowledge of LLM training features and libraries - Checkpointing, Parallelism, Pytorch, Megatron-LM, NCCL</li><li>Experience with HPC schedulers such as Slurm</li><li>Background with Opentelemetry</li></ul>#LI-Hybrid</div>

Senior GPU Cluster Software Engineer