Senior HPC Data Engineer

1 Month ago • 2 Years + • Data Analyst

Job Summary

Job Description

This Senior HPC Data Engineer role involves designing and implementing a data service module for an HPC modeling project. Responsibilities include developing high-performance data storage and retrieval systems using HDF5 (or similar) with parallel I/O capabilities, ensuring tight integration with HPC and visualization workflows, optimizing I/O for CPU/GPU workflows, implementing caching and compression strategies, designing data structures for 3D grid data, ensuring data integrity, developing test cases, conducting benchmarking, and documenting the module. The ideal candidate will have strong Python and HDF5 expertise, experience with parallel I/O frameworks, and skills in performance optimization. SQL and RDMS knowledge are also beneficial. The role requires collaboration with development and visualization teams.
Must have:
  • HDF5 expertise & parallel I/O
  • Python proficiency
  • Data service design & implementation for HPC
  • Performance optimization skills
  • Experience with large datasets
Good to have:
  • C++/Rust experience
  • Knowledge of orchestration tools (Airflow, etc.)
  • Familiarity with scientific computing data formats
Perks:
  • Flexible working format
  • Competitive salary
  • Personalized career growth
  • Professional development tools
  • Education reimbursement
  • Corporate events

Job Details

About our customer:

Our client is a global company specializing in software development and consulting who combines science and technology with deep industry expertise to solve complex subsurface and surface challenges in the evolving energy sector.
The client headquartered is located in Canada, with offices around the world. The company serves organizations globally, providing cutting-edge software technology and unparalleled customer support.

Key Software Solutions:
Our client offers a range of reservoir simulation software, including:

  • IMEX – A black oil simulator for primary, secondary, and tertiary recovery processes.
  • GEM – An advanced simulator for compositional, chemical, and unconventional reservoir modeling.
  • STARS – The industry standard for thermal and advanced recovery processes.
  • CMOST – An intelligent optimization and analysis tool that integrates statistical analysis, machine learning, and unbiased data interpretation to determine optimal reservoir solutions.


The company invests in research and development, continuously improving its products and delivering state-of-the-art solutions for energy modeling and optimization.

About the Role:

We are looking for an experienced Data Service Module Engineer to develop and deploy the data service module for the HPC modeling project. This role focuses on implementing high-performance data storage and retrieval systems using HDF5 or similar, with parallel and concurrent I/O capabilities. The ideal candidate will have expertise in designing scalable data services optimized for HPC or distributed workflows, ensuring low latency and high throughput.

 

Key Responsibilities:

  • Design and implement the data service module using HDF5 for efficient data storage and retrieval.
  • Develop parallel and concurrent I/O mechanisms to optimize performance for large-scale datasets.
  • Ensure the module is tightly integrated with HPC and visualization workflows.
  • Optimize I/O operations for CPU/GPU-based workflows to minimize bottlenecks.
  • Implement caching, compression, and other strategies to enhance performance.
  • Design data structures and schemas suitable for storing 3D grid data and other simulation outputs.
  • Ensure data integrity and consistency during concurrent read/write operations.
  • Develop and execute test cases to validate module performance and reliability under various load conditions.
  • Conduct benchmarking to ensure scalability across different hardware configurations.
  • Document the architecture, APIs, and usage guidelines for the data service module.
  • Provide technical support to the development and visualization teams for data integration.

 

Requirements:

  • Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related fields.
  • 2+ years of experience in developing and deploying data services for HPC or similar systems.
  • Proven expertise in Big Data
  • Proven expertise with HDF5 or similar, in parallel I/O operations. Equivalent experience in distributed systems is also applicable.
  • Programming: Strong proficiency in Python. An additional experience with C++ and/or Rust is a nice to have and it will be considered a plus
  • HDF5 Expertise: In-depth knowledge of HDF5 APIs and advanced features like parallel HDF5.
  • Parallel I/O: Experience with MPI I/O, POSIX I/O, or similar frameworks for concurrent/parallel data access.
  • Performance Optimization: Skills in profiling and optimizing I/O operations for large datasets.
  • Proficiency in SQL and experience with any RDMS
  • Might be a plus: knowledge of at least one orchestration and scheduling tool, for example, Airflow, Prefect, Dagster, etc.
  • Strong problem-solving skills and ability to work in a multidisciplinary team.
  • Excellent communication skills for cross-team collaboration and documentation.

 

Preferred Qualifications:

  • Familiarity with data formats used in scientific computing, 3D visualization, and simulation workflows.

We offer*:

  • Flexible working format - remote, office-based or flexible
  • A competitive salary and good compensation package
  • Personalized career growth
  • Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
  • Active tech communities with regular knowledge sharing
  • Education reimbursement
  • Memorable anniversary presents
  • Corporate events and team buildings
  • Other location-specific benefits

*not applicable for freelancers

Similar Jobs

ByteDance - Senior Software Development Engineer - Distributed NoSQL Database Systems

ByteDance

San Jose, California, United States (On-Site)
4 Months ago
Google - Technical Lead Manager, Android Kernel, Android Systems

Google

Kraków, Lesser Poland Voivodeship, Poland (On-Site)
1 Month ago
Argus Labs - Sr. Software Engineer (Infrastructure/Backend)

Argus Labs

Indonesia (Remote)
2 Months ago
gnosis - Solidity Engineer

gnosis

(Remote)
1 Month ago
Hasura - Senior / Staff Software Engineer (Architect) - Cloud Infrastructure

Hasura

Bengaluru, Karnataka, India (Hybrid)
7 Months ago
Magna International - Data Scientist

Magna International

Bengaluru, Karnataka, India (On-Site)
8 Months ago
Scorewarrior - Product Analyst - Game Security

Scorewarrior

Limassol, Limassol, Cyprus (On-Site)
1 Month ago
Arrise Solutions (India)   - Data Scientist - Recommender S/m's

Arrise Solutions (India)

Hyderabad, Telangana, India (On-Site)
8 Months ago
Sporty Group - Data Engineer

Sporty Group

(Remote)
2 Months ago
Netflix - Analytics Engineer (L4) - Acquisition

Netflix

Los Gatos, California, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Mozilla - Credential Management Student Worker

Mozilla

(Remote)
1 Month ago
Meta - Production Engineering

Meta

Cambridge, Massachusetts, United States (Hybrid)
6 Months ago
Discord - Staff Software Engineer, Media Infrastructure

Discord

San Francisco, California, United States (On-Site)
4 Months ago
Aerospike - Senior Software Engineer, Client (Java + Go)

Aerospike

(Remote)
1 Month ago
Animoca Brands - Web3 Engineer

Animoca Brands

Hong Kong, Hong Kong (Hybrid)
2 Months ago
Polygon Labs - Senior Software Engineer

Polygon Labs

(Remote)
2 Months ago
SimpliSafe - Sr Software Engineer

SimpliSafe

Boston, Massachusetts, United States (Hybrid)
1 Month ago
Coda - Senior/Staff Software Engineer

Coda

Taipei City, Taiwan (Hybrid)
1 Year ago
N-iX - Senior C++ Engineer (High Performance Computing)

N-iX

United Kingdom (Remote)
3 Months ago
Meta - Production Engineering

Meta

Austin, Texas, United States (Hybrid)
6 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Argentina

Haleon - Temporary Quality Control Analyst

Haleon

Buenos Aires, Buenos Aires, Argentina (On-Site)
3 Weeks ago
Room 8 Studio - Realistic 3D Characters Artist, Project Lead Senior

Room 8 Studio

Argentina (On-Site)
3 Months ago
Big Time Studios - Sr 3D Environment Artist

Big Time Studios

Buenos Aires, Buenos Aires, Argentina (Remote)
1 Year ago
N-iX - Senior C++ Engineer (High Performance Computing)

N-iX

Argentina (Remote)
2 Months ago
CloudLinux - Director of Enterprise Sales

CloudLinux

Buenos Aires, Buenos Aires, Argentina (Remote)
2 Months ago
PwC - Manager - Strategy& e Inteligencia Artificial

PwC

Buenos Aires, Buenos Aires, Argentina (On-Site)
4 Months ago
Devrev - Revenue: Sales Development Representative

Devrev

Buenos Aires, Buenos Aires, Argentina (On-Site)
1 Month ago
Haleon - Regional Revenue Management Specialist (Latam)

Haleon

Buenos Aires, Buenos Aires, Argentina (On-Site)
1 Month ago
Neolytix - Healthcare - Bilingual Medical Virtual Assistant

Neolytix

Buenos Aires, Buenos Aires, Argentina (On-Site)
3 Weeks ago
N-iX - Senior DevOps Engineer

N-iX

Argentina (Remote)
2 Months ago

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

Meta - Data Engineer, Product Analytics

Meta

New York, New York, United States (On-Site)
6 Months ago
Millennium - Data Engineer

Millennium

Bengaluru, Karnataka, India (On-Site)
6 Months ago
SciPlay - Business Intelligence Developer

SciPlay

Tel Aviv-Yafo, Tel Aviv District, Israel (On-Site)
5 Months ago
Google - Risk Data Pipeline Analyst

Google

São Paulo, State Of São Paulo, Brazil (On-Site)
1 Month ago
Dream Sports - Software Development Engineer 3 - Machine Learning & Data Platform

Dream Sports

Mumbai, Maharashtra, India (On-Site)
3 Months ago
Animoca Brands - Tokenomics Analyst

Animoca Brands

(On-Site)
2 Months ago
Paytm - Manager - Business Analyst - Offine Merchants- QR

Paytm

Noida, Uttar Pradesh, India (On-Site)
5 Months ago
PwC - Senior Associate

PwC

Bhopal, Madhya Pradesh, India (On-Site)
8 Months ago
PwC - Senior Data Scientist

PwC

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)
8 Months ago

Get notifed when new similar jobs are uploaded