Senior SRE Software Engineer, Storage and Data

SRE at NVIDIA ensures that our DGX Cloud platform continues to be reliable and performant to meet the needs of our users. You will play a critical role in ensuring the reliability, availability, and performance of storage infrastructures for NVIDIA DGX GPU cloud platforms. To collaborate with cross-functional teams to design, build, and maintain scalable and fault-tolerant storage solutions that support our mission-critical applications and services. Your expertise in storage systems and reliability engineering will be instrumental in minimizing downtime, improving system efficiency, and enhancing the overall user experience.

SRE is also a mindset and a set of engineering approaches to running efficient production systems, with a focus on eliminating manual work through modern automation practices and performance tuning. We promote self-direction to work on meaningful projects while striving to build an environment that provides the support and mentorship needed to learn and grow.

What You Will Be Doing:

• Develop strategies to ensure the reliability and availability of storage systems, including redundancy, failover, and disaster recovery plans.
• Continuously analyze and fine-tune storage systems for optimal performance, including throughput optimization, caching, and latency reduction. Identify and resolve performance bottlenecks to enhance overall system efficiency.
• Develop and maintain automation scripts and tools to streamline storage provisioning, configuration, and maintenance tasks.
• Implement monitoring and alerting systems to proactively identify and address issues.
• Participate in on-call rotation to respond to storage-related incidents promptly conduct root cause analysis of outages and implement preventive measures.
• Collaborate with cross-functional teams, including Compute SRE, development, and networking, to ensure seamless integration of large-scale storage solutions.
• Work with AI/ML workloads to capture and correlate behavior in large clusters and workflows, which are otherwise hard to understand.

What We Need To See:

• BS degree in Computer Science or related technical field involving coding (e.g., physics or mathematics), with 5+ years equivalent practical experience.
• Proven experience in storage system administration and site reliability engineering.
• Experience with Git, RESTFul API, Linux service operation, networking, complexity analysis, AWS S3, software design, and maintaining large-scale Linux based systems.
• Experience in one or more of the following languages: Ansible, Bash, Python, Go, YAML, Java
• Good knowledge of infrastructure configuration management tools like Ansible, Chef, Puppet, and Terraform.
• Experience in using observability and tracing-related tools like InfluxDB, Prometheus, and Elastic(OpenSearch) stack, Grafana. 

Ways to stand out from the crowd:

• Experience with storage solutions like: OpenStack Swift(object), AWS S3(object), DDN, Lustre.
• Strong Linux and network troubleshooting skills by running various commands and tools.
• Demonstrated experience in having an SRE mindset, customer-first approach, and focus on customer satisfaction and passion for ensuring customer success..
• Interest in crafting, analyzing, and fixing large-scale distributed systems. Strong debugging skills with a systematic problem-solving approach to identify complex problems.
• Experience in using or running large private and public cloud systems based on Kubernetes, OpenStack, and Docker.