Senior HPC Data Engineer

2 Weeks ago • 3-3 Years • Data Analyst

Job Summary

Job Description

This role involves developing and deploying a high-performance data service module for an HPC modeling project. Responsibilities include designing and implementing the module using HDF5 for efficient data storage and retrieval, developing parallel and concurrent I/O mechanisms, optimizing I/O operations for CPU/GPU workflows, designing data structures for 3D grid data, ensuring data integrity, developing and executing test cases, and documenting the module's architecture and APIs. The ideal candidate will have strong proficiency in C++, Python, GoLang, or Fortran; in-depth knowledge of HDF5 APIs and parallel I/O; and experience optimizing I/O operations for large datasets. Experience with MPI I/O, POSIX I/O, or similar frameworks is also required.
Must have:
  • HDF5 expertise
  • Parallel I/O experience
  • Performance optimization skills
  • C++, Python, GoLang, or Fortran proficiency
  • Data service development for HPC
Good to have:
  • SQL and RDMS experience
  • Orchestration tools (Airflow, Prefect, etc.)
  • Familiarity with scientific computing data formats
Perks:
  • Flexible working format
  • Competitive salary
  • Personalized career growth
  • Professional development tools
  • Education reimbursement
  • Corporate events

Job Details

About the Role:

We are looking for an experienced Data Service Module Engineer to develop and deploy the data service module for the HPC modeling project. This role focuses on implementing high-performance data storage and retrieval systems using HDF5 or similar, with parallel and concurrent I/O capabilities. The ideal candidate will have expertise in designing scalable data services optimized for HPC or distributed workflows, ensuring low latency and high throughput.

 

Key Responsibilities:

1.Module Development and Deployment:

  • Design and implement the data service module using HDF5 for efficient data storage and retrieval.
  • Develop parallel and concurrent I/O mechanisms to optimize performance for large-scale datasets.
  • Ensure the module is tightly integrated with HPC and visualization workflows.


2.Performance Optimization:

  • Optimize I/O operations for CPU/GPU-based workflows to minimize bottlenecks.
  • Implement caching, compression, and other strategies to enhance performance.


3.Data Management:

  • Design data structures and schemas suitable for storing 3D grid data and other simulation outputs.
  • Ensure data integrity and consistency during concurrent read/write operations.


4.Testing and Validation:

  • Develop and execute test cases to validate module performance and reliability under various load conditions.
  • Conduct benchmarking to ensure scalability across different hardware configurations.


5.Documentation and Support:

  • Document the architecture, APIs, and usage guidelines for the data service module.
  • Provide technical support to the development and visualization teams for data integration.

 

Qualifications:

1.Education:

  • Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related fields.

 

2.Experience:

  • 3+ years of experience in developing and deploying data services for HPC or similar systems.
  • Proven expertise with HDF5 or similar, in parallel I/O operations. Equivalent experience in distributed systems is also applicable.

 

3.Technical Skills:

  • Programming: Strong proficiency in (at least one): C++, Python, GoLang, or Fortran.
  • HDF5 Expertise: In-depth knowledge of HDF5 APIs and advanced features like parallel HDF5.
  • Parallel I/O: Experience with MPI I/O, POSIX I/O, or similar frameworks for concurrent/parallel data access.
  • Performance Optimization: Skills in profiling and optimizing I/O operations for large datasets.
  • Proficiency in SQL and experience with any RDMS
  • Might be a plus: knowledge of at least one orchestration and scheduling tool, for example, Airflow, Prefect, Dagster, etc.

 

4.Soft Skills:

  • Strong problem-solving skills and ability to work in a multidisciplinary team.
  • Excellent communication skills for cross-team collaboration and documentation.

 

Preferred Qualifications:

  • Familiarity with data formats used in scientific computing, 3D visualization, and simulation workflows.

We offer:

  • Flexible working format - remote, office-based or flexible
  • A competitive salary and good compensation package
  • Personalized career growth
  • Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
  • Active tech communities with regular knowledge sharing
  • Education reimbursement
  • Memorable anniversary presents
  • Corporate events and team buildings
  • Other location-specific benefits

Similar Jobs

ByteDance - Site Reliability Engineer, Traffic Infrastructure

ByteDance

Singapore (On-Site)
3 Months ago
Krafton  - [인조이스튜디오(신규 자회사)] Sr. Technical Artist (8년 이상)

Krafton

Seoul, South Korea (On-Site)
2 Months ago
Vigaet - Internship - Computer Graphics Programmer

Vigaet

Bengaluru, Karnataka, India (On-Site)
9 Months ago
Microsoft - Principal Software Engineering Manager - Mobile

Microsoft

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Meta - Software Engineering Manager, Product Infrastructure

Meta

Redmond, Washington, United States (Remote)
3 Months ago
Sportskeeda - Data Analyst

Sportskeeda

India (Remote)
2 Weeks ago
Paypal - Database Marketing Manager

Paypal

Shanghai, Shanghai, China (On-Site)
4 Months ago
The Walt Disney Company - Sr Machine Learning Engineer

The Walt Disney Company

Santa Monica, California, United States (On-Site)
3 Months ago
N-iX - Data Engineer (Databricks) (#2168)

N-iX

Romania (Remote)
3 Months ago
Truecaller - Senior Data Analyst

Truecaller

Stockholm, Stockholm County, Sweden (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

The Walt Disney Company - Sr Software Engineer (Rust Developer)

The Walt Disney Company

San Francisco, California, United States (On-Site)
3 Months ago
Visual Concepts - Senior Server Engineer, NBA 2K

Visual Concepts

Novato, California, United States (On-Site)
2 Months ago
Keywords Studios (Player Support) - Head of Programming - Americas

Keywords Studios (Player Support)

Canada (Remote)
1 Week ago
Alp Consulting  - Multimedia

Alp Consulting

Bengaluru, Karnataka, India (On-Site)
9 Months ago
Microsoft - Silicon Engineering: Internship Opportunities

Microsoft

Penang, Malaysia (On-Site)
1 Month ago
Gameloft - [D&D PC-Console ] Directeur(trice) artistique technique- Technical art director

Gameloft

Montreal, Quebec, Canada (Hybrid)
5 Months ago
NVIDIA - Manager, Design Verification

NVIDIA

Hsinchu, Hsinchu City, Taiwan (Hybrid)
3 Weeks ago
Epic Games - Senior Engine Programmer, Framework

Epic Games

Montreal, Quebec, Canada (On-Site)
1 Month ago
NVIDIA - Senior Math Libraries Engineer - Dense Linear Algebra

NVIDIA

California, United States (Hybrid)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in United Kingdom

ByteDance - Linux System Engineer

ByteDance

London, England, United Kingdom (On-Site)
1 Month ago
CloudHire - Frontend Testing Engineer

CloudHire

England, United Kingdom (Hybrid)
2 Months ago
Cloud Imperium Games - Lead Gameplay Animation Programmer

Cloud Imperium Games

Manchester, England, United Kingdom (On-Site)
2 Months ago
PTW - Junior Post-Production Coordinator

PTW

England, United Kingdom (On-Site)
1 Week ago
Axinous - Senior Sales Engineer - Major Accounts UK

Axinous

United Kingdom (Remote)
5 Days ago
Electronic Arts - Senior Experience Designer - FC

Electronic Arts

Southam, England, United Kingdom (On-Site)
3 Weeks ago
Activision - Lead Systems Designer

Activision

Middleton, England, United Kingdom (On-Site)
5 Months ago
Rank group - Team Leader - Food & Beverage

Rank group

Gateshead, England, United Kingdom (On-Site)
3 Months ago
IGN - Editorial Director, GamesIndustry.biz

IGN

London, England, United Kingdom (Hybrid)
2 Months ago
Assystems - NEC 4 Supervisor

Assystems

Helensburgh, Scotland, United Kingdom (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

GoTo Group - Senior Data Analyst

GoTo Group

Bengaluru, Karnataka, India (On-Site)
4 Months ago
HP - Privacy Analyst  (Houston/Austin/Fort Collins)

HP

Spring, Texas, United States (On-Site)
5 Months ago
Evernorth Health Services - Software Engineering Advisor [T500-10128]

Evernorth Health Services

Hyderabad, Telangana, India (On-Site)
4 Months ago
Playrix - Senior Data Analyst (Attribution)

Playrix

Montenegro (Remote)
4 Months ago
PwC - IN-Manager_Delivery Manager_Data & Analytics_Advisory_Mumbai

PwC

Mumbai, Maharashtra, India (On-Site)
4 Months ago
GoMotive - Data Scientist, Credit Risk

GoMotive

United States (Remote)
6 Days ago
The Walt Disney Company - Sr Machine Learning Engineer

The Walt Disney Company

Santa Monica, California, United States (On-Site)
3 Months ago
Inxite Out - Senior Data Scientist - NLP

Inxite Out

Bengaluru, Karnataka, India (On-Site)
8 Months ago
Myntra - Senior Data Engineer

Myntra

Bengaluru, Karnataka, India (On-Site)
4 Months ago

Get notifed when new similar jobs are uploaded