Senior HPC Data Engineer

2 Months ago • 3-3 Years • Data Analyst

Job Summary

Job Description

This role involves developing and deploying a high-performance data service module for an HPC modeling project. Responsibilities include designing and implementing the module using HDF5 for efficient data storage and retrieval, developing parallel and concurrent I/O mechanisms, optimizing I/O operations for CPU/GPU workflows, designing data structures for 3D grid data, ensuring data integrity, developing and executing test cases, and documenting the module's architecture and APIs. The ideal candidate will have strong proficiency in C++, Python, GoLang, or Fortran; in-depth knowledge of HDF5 APIs and parallel I/O; and experience optimizing I/O operations for large datasets. Experience with MPI I/O, POSIX I/O, or similar frameworks is also required.
Must have:
  • HDF5 expertise
  • Parallel I/O experience
  • Performance optimization skills
  • C++, Python, GoLang, or Fortran proficiency
  • Data service development for HPC
Good to have:
  • SQL and RDMS experience
  • Orchestration tools (Airflow, Prefect, etc.)
  • Familiarity with scientific computing data formats
Perks:
  • Flexible working format
  • Competitive salary
  • Personalized career growth
  • Professional development tools
  • Education reimbursement
  • Corporate events

Job Details

About the Role:

We are looking for an experienced Data Service Module Engineer to develop and deploy the data service module for the HPC modeling project. This role focuses on implementing high-performance data storage and retrieval systems using HDF5 or similar, with parallel and concurrent I/O capabilities. The ideal candidate will have expertise in designing scalable data services optimized for HPC or distributed workflows, ensuring low latency and high throughput.

 

Key Responsibilities:

1.Module Development and Deployment:

  • Design and implement the data service module using HDF5 for efficient data storage and retrieval.
  • Develop parallel and concurrent I/O mechanisms to optimize performance for large-scale datasets.
  • Ensure the module is tightly integrated with HPC and visualization workflows.


2.Performance Optimization:

  • Optimize I/O operations for CPU/GPU-based workflows to minimize bottlenecks.
  • Implement caching, compression, and other strategies to enhance performance.


3.Data Management:

  • Design data structures and schemas suitable for storing 3D grid data and other simulation outputs.
  • Ensure data integrity and consistency during concurrent read/write operations.


4.Testing and Validation:

  • Develop and execute test cases to validate module performance and reliability under various load conditions.
  • Conduct benchmarking to ensure scalability across different hardware configurations.


5.Documentation and Support:

  • Document the architecture, APIs, and usage guidelines for the data service module.
  • Provide technical support to the development and visualization teams for data integration.

 

Qualifications:

1.Education:

  • Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related fields.

 

2.Experience:

  • 3+ years of experience in developing and deploying data services for HPC or similar systems.
  • Proven expertise with HDF5 or similar, in parallel I/O operations. Equivalent experience in distributed systems is also applicable.

 

3.Technical Skills:

  • Programming: Strong proficiency in (at least one): C++, Python, GoLang, or Fortran.
  • HDF5 Expertise: In-depth knowledge of HDF5 APIs and advanced features like parallel HDF5.
  • Parallel I/O: Experience with MPI I/O, POSIX I/O, or similar frameworks for concurrent/parallel data access.
  • Performance Optimization: Skills in profiling and optimizing I/O operations for large datasets.
  • Proficiency in SQL and experience with any RDMS
  • Might be a plus: knowledge of at least one orchestration and scheduling tool, for example, Airflow, Prefect, Dagster, etc.

 

4.Soft Skills:

  • Strong problem-solving skills and ability to work in a multidisciplinary team.
  • Excellent communication skills for cross-team collaboration and documentation.

 

Preferred Qualifications:

  • Familiarity with data formats used in scientific computing, 3D visualization, and simulation workflows.

We offer:

  • Flexible working format - remote, office-based or flexible
  • A competitive salary and good compensation package
  • Personalized career growth
  • Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
  • Active tech communities with regular knowledge sharing
  • Education reimbursement
  • Memorable anniversary presents
  • Corporate events and team buildings
  • Other location-specific benefits

Similar Jobs

Google - Software Engineer III, Infrastructure, Google Cloud NetInfra

Google

Madison, Wisconsin, United States (On-Site)
1 Week ago
Every matrix - Middle Java Developer

Every matrix

Bucharest, Bucharest, Romania (Hybrid)
4 Weeks ago
Google - Staff Software Engineer, Infrastructure, Core

Google

New York, New York, United States (On-Site)
6 Days ago
Warner Bros Games - Staff Software Engineer - Data Platform

Warner Bros Games

Hyderabad, Telangana, India (Hybrid)
3 Weeks ago
Google - Senior Software Engineer, Core

Google

Hyderabad, Telangana, India (On-Site)
6 Days ago
The Walt Disney Company - Data Governance Engineer

The Walt Disney Company

New York, New York, United States (On-Site)
2 Months ago
BrightEdge - Database Engineer

BrightEdge

Hyderabad, Telangana, India (Remote)
6 Months ago
The Walt Disney Company - Senior Data Analyst

The Walt Disney Company

Santa Monica, California, United States (On-Site)
1 Week ago
Globalization Partners - Business Intelligence Analyst

Globalization Partners

(Remote)
3 Months ago
Next Level Business Services - Markit EDM

Next Level Business Services

Pittsburgh, Pennsylvania, United States (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Google - Senior Software Engineer, Machine Learning, Applied AI

Google

Sunnyvale, California, United States (On-Site)
1 Week ago
Google - ASIC Power Architect, Silicon

Google

New Taipei, New Taipei City, Taiwan (On-Site)
1 Week ago
NVIDIA - Senior GPU Cluster Software Engineer

NVIDIA

Shanghai, Shanghai, China (On-Site)
3 Months ago
Canva - Senior Frontend Engineer - Organising Content

Canva

Melbourne, Victoria, Australia (Remote)
3 Weeks ago
Netflix - Network Architecture Intern, Summer 2025

Netflix

Los Gatos, California, United States (On-Site)
5 Months ago
Google - Senior Software Engineer, Mobile, Android

Google

Mountain View, California, United States (On-Site)
6 Days ago
Canva - Senior Frontend Engineer - Apps API Platform

Canva

Melbourne, Victoria, Australia (Remote)
4 Weeks ago
Anavation - Senior Software Engineer

Anavation

San Antonio, Texas, United States (Remote)
1 Day ago
The Walt Disney Company - Senior Data Analyst

The Walt Disney Company

Santa Monica, California, United States (On-Site)
1 Week ago
Canva - Frontend Software Engineer - Internationalization

Canva

Beijing, Beijing, China (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in United Kingdom

Ubisoft - Player Journey Mapper

Ubisoft

Newcastle Upon Tyne, England, United Kingdom (On-Site)
1 Week ago
PlayStation Global - Staff HRIS Analyst: Talent Management & Workday

PlayStation Global

London, England, United Kingdom (On-Site)
4 Weeks ago
Inspired Entertainment - Customer Service Assistant

Inspired Entertainment

Skipsea, England, United Kingdom (On-Site)
1 Week ago
The Walt Disney Company - Supervisor, DTC Content Fulfillment

The Walt Disney Company

London, England, United Kingdom (On-Site)
1 Month ago
Maverick Games - Principal Game Designer

Maverick Games

Warwick, England, United Kingdom (Hybrid)
1 Week ago
Dream Games - Brand Marketing Specialist (New Grad)

Dream Games

London, England, United Kingdom (On-Site)
3 Months ago
DNEG - Character TD (DNEG ANIMATION)

DNEG

London, England, United Kingdom (On-Site)
11 Months ago
Sporty Group - Unity Game Developer

Sporty Group

Bournemouth, England, United Kingdom (Hybrid)
1 Day ago
Kwalee - Product Marketing Manager

Kwalee

Royal Leamington Spa, England, United Kingdom (On-Site)
1 Month ago
Kwalee - Junior Product Marketing Manager

Kwalee

Royal Leamington Spa, England, United Kingdom (On-Site)
3 Weeks ago

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

Ness Digital - Big Data Engineer

Ness Digital

Timișoara, Timiș, Romania (Remote)
4 Weeks ago
ION - Data Associate - KYC6

ION

Budapest, Hungary (On-Site)
6 Months ago
PwC - Senior Associate

PwC

Bhopal, Madhya Pradesh, India (On-Site)
6 Months ago
DraftKings - Customer Trading Analyst

DraftKings

Plovdiv, Plovdiv Province, Bulgaria (On-Site)
4 Weeks ago
The Walt Disney Company - Data Analyst Internship - Finance Division

The Walt Disney Company

Serris, Île-de-France, France (On-Site)
1 Week ago
The Walt Disney Company - Data Engineer, VX Analytics

The Walt Disney Company

Santa Monica, California, United States (On-Site)
1 Week ago
ION - Data Associate - Wealthmonitor

ION

Budapest, Hungary (On-Site)
6 Months ago
NinjaVan - Manager, Partnership (Key Account  Management)

NinjaVan

Subang Jaya, Selangor, Malaysia (On-Site)
6 Months ago
DWS Group - Data Scientist

DWS Group

Mumbai, Maharashtra, India (Hybrid)
5 Months ago
Riot Games - Insights Analyst II, Wild Rift

Riot Games

Singapore (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded