Senior HPC AI Cluster Engineer

NVIDIA is looking for an experienced HPC Engineer to join the E2E software verification HPC/AI Infrastructure team. We are building supercomputers and HPC clusters based on groundbreaking technologies. We are looking for an outstanding architect for a senior HPC, be a key player to the most exciting computing hardware and software to contribute to the latest breakthroughs in artificial intelligence and GPU computing. Provide insights on at-scale system design and tuning mechanisms for large-scale compute runs.

You will work with the latest Accelerated computing and Deep Learning software and hardware platforms, and with many scientific researchers, developers, and customers to craft improved workflows and develop new, leading differentiated solutions. You will interact with HPC, OS, GPU compute, and systems specialist to architect, develop and bring up large scale performance platforms. Does this sound like you? If so, we would love to hear from you!

What you will be doing:

• Designing, implementing and maintaining large scale HPC/AI clusters with monitoring, logging and alerting

• Managing Linux job/workload schedules and orchestration tools

• Developing and maintaining continuous integration and delivery pipelines

• Developing tooling to automate deployment and management of large-scale infrastructure environments, to automate operational monitoring and alerting, and to enable self-service consumption of resources

• Deploying monitoring solutions for the servers, network and storage

• Troubleshooting and fixing, bottom up from bare metal, operating system, software stack and application level

• Being a technical resource, developing, re-defining and documenting standard methodologies to share with internal teams

• Supporting Research & Development activities and engaging in POCs/POVs for future improvements

What we need to see:

• Bachelor's Degree in Computer Science, Engineering, or a related field; or equivalent experience

• 5+ years of experience

• Knowledge of HPC and AI solution technologies from CPU’s and GPU’s to high speed interconnects and supporting software

• Experience with job scheduling workloads and orchestration tools such as Slurm, K8s

• Excellent knowledge of Windows and Linux (Redhat/CentOS and Ubuntu) networking (sockets, firewalls, iptables, wireshark, etc.) and internals, ACLs and OS level security protection and common protocols e.g. TCP, DHCP, DNS, etc.

• Experience with multiple storage solutions such as Lustre, GPFS, zfs and xfs. Familiarity with newer and emerging storage technologies.

• Python programming and bash scripting experience.

• Comfortable with automation and configuration management tools such as Jenkins, Ansible, Puppet/chef

• Deep knowledge of Networking Protocols like InfiniBand, Ethernet

• Deep understanding and experience with virtual systems (for example VMware, Hyper-V, KVM, or Citrix)

• Familiarity with cloud computing platforms (e.g. AWS, Azure, Google Cloud)

Ways to stand out from the crowd:

• Knowledge of CPU and/or GPU architecture

• Knowledge of Kubernetes, container related microservice technologies

• Experience with GPU-focused hardware/software (DGX, Cuda)

• Background with RDMA (InfiniBand or RoCE) fabrics

NVIDIA has been redefining computer graphics, PC gaming, and accelerated computing for more than 25 years. We have a unique legacy of innovation that’s fueled by great technology—and amazing people. Today, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what’s never been done before takes vision, innovation, and the world’s best talent.