Senior HPC Data Engineer

2 Months ago • 3-3 Years • Data Analyst

Job Summary

Job Description

This role involves developing and deploying a high-performance data service module for an HPC modeling project. Responsibilities include designing and implementing the module using HDF5 for efficient data storage and retrieval, developing parallel and concurrent I/O mechanisms, optimizing I/O operations for CPU/GPU workflows, designing data structures for 3D grid data, ensuring data integrity, developing and executing test cases, and documenting the module's architecture and APIs. The ideal candidate will have strong proficiency in C++, Python, GoLang, or Fortran; in-depth knowledge of HDF5 APIs and parallel I/O; and experience optimizing I/O operations for large datasets. Experience with MPI I/O, POSIX I/O, or similar frameworks is also required.
Must have:
  • HDF5 expertise
  • Parallel I/O experience
  • Performance optimization skills
  • C++, Python, GoLang, or Fortran proficiency
  • Data service development for HPC
Good to have:
  • SQL and RDMS experience
  • Orchestration tools (Airflow, Prefect, etc.)
  • Familiarity with scientific computing data formats
Perks:
  • Flexible working format
  • Competitive salary
  • Personalized career growth
  • Professional development tools
  • Education reimbursement
  • Corporate events

Job Details

About the Role:

We are looking for an experienced Data Service Module Engineer to develop and deploy the data service module for the HPC modeling project. This role focuses on implementing high-performance data storage and retrieval systems using HDF5 or similar, with parallel and concurrent I/O capabilities. The ideal candidate will have expertise in designing scalable data services optimized for HPC or distributed workflows, ensuring low latency and high throughput.

 

Key Responsibilities:

1.Module Development and Deployment:

  • Design and implement the data service module using HDF5 for efficient data storage and retrieval.
  • Develop parallel and concurrent I/O mechanisms to optimize performance for large-scale datasets.
  • Ensure the module is tightly integrated with HPC and visualization workflows.


2.Performance Optimization:

  • Optimize I/O operations for CPU/GPU-based workflows to minimize bottlenecks.
  • Implement caching, compression, and other strategies to enhance performance.


3.Data Management:

  • Design data structures and schemas suitable for storing 3D grid data and other simulation outputs.
  • Ensure data integrity and consistency during concurrent read/write operations.


4.Testing and Validation:

  • Develop and execute test cases to validate module performance and reliability under various load conditions.
  • Conduct benchmarking to ensure scalability across different hardware configurations.


5.Documentation and Support:

  • Document the architecture, APIs, and usage guidelines for the data service module.
  • Provide technical support to the development and visualization teams for data integration.

 

Qualifications:

1.Education:

  • Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related fields.

 

2.Experience:

  • 3+ years of experience in developing and deploying data services for HPC or similar systems.
  • Proven expertise with HDF5 or similar, in parallel I/O operations. Equivalent experience in distributed systems is also applicable.

 

3.Technical Skills:

  • Programming: Strong proficiency in (at least one): C++, Python, GoLang, or Fortran.
  • HDF5 Expertise: In-depth knowledge of HDF5 APIs and advanced features like parallel HDF5.
  • Parallel I/O: Experience with MPI I/O, POSIX I/O, or similar frameworks for concurrent/parallel data access.
  • Performance Optimization: Skills in profiling and optimizing I/O operations for large datasets.
  • Proficiency in SQL and experience with any RDMS
  • Might be a plus: knowledge of at least one orchestration and scheduling tool, for example, Airflow, Prefect, Dagster, etc.

 

4.Soft Skills:

  • Strong problem-solving skills and ability to work in a multidisciplinary team.
  • Excellent communication skills for cross-team collaboration and documentation.

 

Preferred Qualifications:

  • Familiarity with data formats used in scientific computing, 3D visualization, and simulation workflows.

We offer:

  • Flexible working format - remote, office-based or flexible
  • A competitive salary and good compensation package
  • Personalized career growth
  • Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
  • Active tech communities with regular knowledge sharing
  • Education reimbursement
  • Memorable anniversary presents
  • Corporate events and team buildings
  • Other location-specific benefits

Similar Jobs

Canva - Senior Frontend Engineer - Content Permissions & Sharing

Canva

Auckland, Auckland, New Zealand (Remote)
1 Month ago
Google - Staff Software Engineer, Engineering Productivity, Google Cloud Platforms

Google

Sunnyvale, California, United States (On-Site)
2 Weeks ago
Google - Software Engineer II, Applied AI

Google

Kraków, Lesser Poland Voivodeship, Poland (On-Site)
2 Weeks ago
No brake games - Unity Technical VFX Artist

No brake games

Vilnius, Vilnius County, Lithuania (On-Site)
3 Hours ago
Google - Senior Software Engineer, Full Stack, Guided Support Experience

Google

Pittsburgh, Pennsylvania, United States (On-Site)
2 Days ago
PwC - Associate

PwC

Pune, Maharashtra, India (On-Site)
7 Months ago
Sphere Entertainment Co - Analyst Data Governance

Sphere Entertainment Co

Las Vegas, Nevada, United States (On-Site)
4 Months ago
Netflix - Senior Data Engineer, Commerce Product

Netflix

Mexico City, Mexico City, Mexico (On-Site)
2 Months ago
GoMotive - Senior Strategic Analyst, Product Reliability

GoMotive

United States (Remote)
2 Months ago
Rackspace Technology - QA Lead (Data Engineering) - Modern Data Platform Implementation

Rackspace Technology

Gurugram, Haryana, India (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Google - Software Engineer II, User Protections, Core

Google

Zürich, Zurich, Switzerland (On-Site)
1 Week ago
Google - Software Engineer III, Infrastructure, Core

Google

Sunnyvale, California, United States (On-Site)
1 Week ago
Google - Software Engineer III, Infrastructure, Google Cloud Platform

Google

Sunnyvale, California, United States (On-Site)
2 Weeks ago
Build A Rocket Boy - Senior Engineer

Build A Rocket Boy

(Remote)
1 Month ago
Google - Software Engineer III, Infrastructure, Google Cloud Business Platforms

Google

Sunnyvale, California, United States (On-Site)
2 Weeks ago
Canva - Backend Software Engineer - Internationalization

Canva

Beijing, Beijing, China (Remote)
1 Month ago
Google - Software Engineer II, Site Reliability Engineering, Cloud Logs

Google

Warsaw, Masovian Voivodeship, Poland (On-Site)
2 Weeks ago
Google - Software Engineer III, Chrome

Google

San Francisco, California, United States (On-Site)
2 Weeks ago
Flexra Software - Software Development Manager

Flexra Software

Bengaluru, Karnataka, India (On-Site)
22 Hours ago
Shyft Labs - Data Scientist

Shyft Labs

Toronto, Ontario, Canada (Hybrid)
1 Year ago

Get notifed when new similar jobs are uploaded

Jobs in United Kingdom

DNEG - Mid FX TD (DNEG Animation)

DNEG

London, England, United Kingdom (On-Site)
6 Months ago
Tesla - Sales Leader - Park Royal (London)

Tesla

London, England, United Kingdom (On-Site)
2 Months ago
Larian Studios - Soundscape Designer

Larian Studios

Guildford, England, United Kingdom (On-Site)
6 Months ago
Rebellion - Lead Environment Artist

Rebellion

England, United Kingdom (Hybrid)
1 Month ago
Tencent - Executive Producer - Content & Quality

Tencent

London, England, United Kingdom (On-Site)
1 Month ago
Maverick Games - Senior UI Artist

Maverick Games

Warwick, England, United Kingdom (Hybrid)
4 Months ago
Cloud Imperium Games - Game Designer (Vehicle Specialist)

Cloud Imperium Games

Manchester, England, United Kingdom (On-Site)
1 Month ago
Climax Studios - Lead Level Designer

Climax Studios

Portsmouth, England, United Kingdom (On-Site)
1 Month ago
Every matrix - Java Developer

Every matrix

London, England, United Kingdom (Hybrid)
3 Months ago
Trailer Park - Control Room Assistant

Trailer Park

London, England, United Kingdom (Hybrid)
1 Day ago

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

Dream Sports - Lead ML Scientist

Dream Sports

Mumbai, Maharashtra, India (On-Site)
1 Week ago
BabaEntertainment - BI Developer

BabaEntertainment

California, United States (On-Site)
2 Weeks ago
Epic Games - Analytics Lead (Gameplay)

Epic Games

Cary, North Carolina, United States (On-Site)
3 Months ago
PwC - Senior Associate - D&A - GDC

PwC

Kolkata, West Bengal, India (On-Site)
7 Months ago
Google - Customer Engineer, Looker, Google Cloud

Google

Dublin, County Dublin, Ireland (On-Site)
1 Week ago
Easygo - Data Engineer

Easygo

Melbourne, Victoria, Australia (On-Site)
2 Months ago
Nintendo - Customer Business Analyst (Sales Planning Analyst)

Nintendo

Redmond, Washington, United States (Hybrid)
3 Months ago
Google - Data Scientist, Chrome

Google

Warsaw, Masovian Voivodeship, Poland (On-Site)
2 Weeks ago
Behaviour Interactive - Senior Data Analyst

Behaviour Interactive

Montreal, Quebec, Canada (Hybrid)
2 Weeks ago
Scorewarrior - Data Analyst

Scorewarrior

Limassol, Limassol, Cyprus (On-Site)
3 Weeks ago

Get notifed when new similar jobs are uploaded