Site Reliability Engineer, HPC and LSF

NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It’s a unique legacy of innovation that’s fueled by great technology—and amazing people. Today, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what’s never been done before takes vision, innovation, and the world’s best talent. As an NVIDIAN, you’ll be immersed in a diverse, supportive environment where everyone is inspired to do their best work. Come join the team and see how you can make a lasting impact on the world.
 

As an SRE, you'll collaborate with various teams to improve our infrastructure environment within NVIDIA's Hardware Infrastructure team. You will enable our engineers to have the best environment on the planet to make the most innovative chips in the world. You will work with your team of EDA and software experts to build new infrastructure in an agile environment. You will continuously innovate and improve scalable, reliable, high performance systems and tools to enable the next generation of chips!
 

What you’ll be doing:

• Manage and support workload and resource schedulers in a large-scale HPC environment.

• Automate Everything: Develop automation scripts to automate deployment, configuration management, and operational monitoring.

• Develop solutions for complex computing resource management requirements.

• Extract and leverage grid performance metrics for troubleshooting and performance optimization.

• Troubleshoot Complex Issues: Perform comprehensive troubleshooting from bare metal to application level, ensuring system reliability and efficiency.

• Develop, define and document standard methodologies to share with internal teams.

• Collaborate with domain experts to improve how our chip development process utilizes our infrastructure.

• Directly contribute to the overall quality and improve time to market for our next generation chips.

What we need to see:

• Extensive knowledge with job scheduler administration (e.g. IBM Spectrum LSF or SLURM).

• Proficient in administering Centos/RHEL Linux distributions.

• In depth understating of container technologies like Docker.

• Proficiency in UNIX scripting languages.

• Excellent problem-solving skills, with the ability to analyze complex systems, identify bottlenecks, and implement scalable solutions.

• Excellent communication and teamwork skills, with the ability to work effectively with diverse teams and individuals.

• 10+ years experience in a large, distributed Linux environment.

• BS in Computer Science, similar degree or equivalent experience.

Ways to stand out from the crowd:

• Experience analyzing and tuning performance for a variety of HPC or EDA workloads.

• Solid understanding of cluster configuration managements tools such as Ansible.

• Proficiency in Perl for maintaining legacy automation scripts.

• Deep understanding of distributed system principles.