Staff Software Engineer - Distributed Data Systems

2 Months ago • All levels • Data Analysis

Job Summary

Job Description

As a Staff Software Engineer, you will be responsible for designing, building, and optimizing data clusters to ensure scalability, fault tolerance, and high availability. The role involves focusing on improving Spark, Hadoop, Kubernetes, Delta Lake, and Druid ecosystems. You will also be involved in evolving these open-source projects internally and contributing code upstream. You will also educate and grow Adyen’s internal knowledge in these topics, collaborating with platform engineers and users.
Must have:
  • Scaling and tuning large deployments of Spark-on-k8s and Spark-on-Hadoop
  • Hadoop and the HDFS protocol
  • Designing and tuning shuffle-heavy systems on yarn or k8s
  • Experience with lakehouse file formats (Delta, Iceberg, Hudi)
  • Experience with OLAP technologies (Clickhouse, Druid, Pinot, Doris)
  • Open-source contributions in relevant technologies
  • Strong communication skills and ability to work in a team
  • Ability to troubleshoot and resolve issues in production environments
Good to have:
  • Experience with next-generation and multi-modal data formats
  • Building self-service stateful platforms
  • Experience with open-source S3 alternatives
  • Experience with native or accelerated runtimes for Spark

Job Details

This is Adyen

Adyen provides payments, data, and financial products in a single solution for customers like Meta, Uber, H&M, and Microsoft - making us the financial technology platform of choice. At Adyen, everything we do is engineered for ambition. 

For our teams, we create an environment with opportunities for our people to succeed, backed by the culture and support to ensure they are enabled to truly own their careers. We are motivated individuals who tackle unique technical challenges at scale and solve them as a team. Together, we deliver innovative and ethical solutions that help businesses achieve their ambitions faster.

Staff Software Engineer - Distributed Data Systems

Adyen hosts a significant footprint in both scale and variety of distributed systems. This covers distributed compute (Spark, Trino, Flink), distributed databases (Cassandra, Druid), and distributed file/object storage (HDFS, Ceph Object Gateway). These technologies are offered as-a-service internally, towards product engineering, to support them in building and scaling world-class products.

We’re looking for an expert with deep knowledge in distributed systems, to both improve operations/scalability of existing offerings, as well as introduce and mature new ones. The initial focus will be scaling and tuning our Hadoop and Spark infrastructure, in addition to iterating on our OLAP platform, where we currently use Apache Druid. Over time, we expect this expertise to be useful in a wider range of distributed systems internally.

What you’ll do

You’ll be asked to both build new as-a-service offerings, and improve the existing ones. This role is perfect for you if you're passionate about one or all of the following:

  • Design, build, and optimize data clusters to ensure scalability, fault tolerance, and high availability. Covering both batch and streaming workloads.
  • Focus on improving the Spark, Hadoop, Kubernetes, Deltalake, and Druid ecosystems internally.
  • Evolve these open-source projects internally, with the intention of contributing this code upstream.
  • Educate and grow Adyen’s internal knowledge in these topics. Covering both peer platform engineers, and the platform users.

Who you are

Must have experience in:

  • Scaling and tuning large deployments of Spark-on-k8s and Spark-on-Hadoop
  • Hadoop and the HDFS protocol
  • Designing and tuning shuffle heavy systems, on yarn, or on k8s via remote shuffle services
  • One of the lakehouse file formats (Delta, Iceberg, Hudi)
  • OLAP technologies covering at least one of Clickhouse, Apache Druid, Apache Pinot, or Apache Doris.
  • Open-source contributions in one of the must-have technologies, or other common ones (e.g. Kafka, Cassandra, Trino, etc)
  • Team player with strong communication skills
  • Ability to work closely with diverse stakeholders you enable (analysts, data scientists, data engineers, etc.) and depend upon (infrastructure, security, etc).
  • Demonstrated ability to troubleshoot and resolve issues in large-scale, production environments with distributed systems.

    Nice-to-have experience in:

  • Next generation and multi-modal data formats (e.g. LanceDB)
  • Building self-service stateful platforms
  • Open-source S3 alternatives (e.g. ceph, minio, etc)
  • Native or accelerated runtimes for Spark (Apache DataFusion Comet, Apache Gluten, Nvidia RAPIDS, etc)

Our Diversity, Equity and Inclusion commitments 

Our unique approach is a product of our diverse perspectives. This diversity of backgrounds and cultures is essential in helping us maintain our momentum. Our business and technical challenges are unique, and we need as many different voices as possible to join us in solving them - voices like yours. No matter who you are or where you’re from, we welcome you to be your true self at Adyen. 

Studies show that women and members of underrepresented communities apply for jobs only if they meet 100% of the qualifications. Does this sound like you? If so, Adyen encourages you to reconsider and apply. We look forward to your application!

What’s next?

Ensuring a smooth and enjoyable candidate experience is critical for us. We aim to get back to you regarding your application within 5 business days. Our interview process tends to take about 4 weeks to complete, but may fluctuate depending on the role. Learn more about our hiring process here. Don’t be afraid to let us know if you need more flexibility.

This role is based out of our Amsterdam office. We are an office-first company and value in-person collaboration; we do not offer remote-only roles.

Similar Jobs

Com2us Corporation - Application Pool

Com2us Corporation

Berlin, Berlin, Germany (On-Site)
12 Months ago
31st Union - Senior Development Manager

31st Union

San Mateo, California, United States (Hybrid)
2 Months ago
NinjaVan - Field Sales Executive Jakarta (Talent Pool)

NinjaVan

Jakarta, Jakarta, Indonesia (On-Site)
9 Months ago
Adyen - Legal Counsel II

Adyen

Singapore (On-Site)
4 Weeks ago
Dialpad AI - Sales Engineer III

Dialpad AI

San Ramon, California, United States (On-Site)
3 Weeks ago
Token Metrics - Senior Crypto Data Engineer (Global-Remote-Non-US)

Token Metrics

Austin, Texas, United States (Remote)
1 Week ago
Mindstorm studios - Data Analyst

Mindstorm studios

Lahore, Punjab, Pakistan (On-Site)
1 Month ago
KPIT - CTO_ML/DL Data scientist

KPIT

Pune, Maharashtra, India (On-Site)
9 Months ago
Mistral AI - Software Engineer, Data

Mistral AI

Paris, Île-de-France, France (Hybrid)
2 Weeks ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Cadence - Software Engineer (C++, Numerical Analysis, EDA)

Cadence

Boston, Massachusetts, United States (On-Site)
3 Weeks ago
Valeo - Advanced Development Technical Engineer

Valeo

Skawina, Lesser Poland Voivodeship, Poland (On-Site)
2 Months ago
fluence - Engineer, RMDC

fluence

Bengaluru, Karnataka, India (On-Site)
2 Months ago
zeta - Associate Product Manager II

zeta

Bengaluru, Karnataka, India (On-Site)
1 Week ago
Sperasoft - Design Director

Sperasoft

St. Julian's, Malta (Hybrid)
2 Months ago
Wind River - Engineer-Services

Wind River

Bengaluru, Karnataka, India (On-Site)
1 Week ago
Animoca Brands - Senior Associate (Strategy Ops / BD & Partnerships / Investments)

Animoca Brands

Dubai, Dubai, United Arab Emirates (Remote)
5 Months ago
Aesir Interactive - Unreal Engine Programmer Games (Regular/ Senior) (f/m/d)

Aesir Interactive

Munich, Bavaria, Germany (Hybrid)
11 Months ago
design works gaming - Business Development Associate - B2B Gaming

design works gaming

Scottsdale, Arizona, United States (Hybrid)
2 Weeks ago
Clearwater Analytics - AVP, Sales Development Representative - Player Coach

Clearwater Analytics

Hong Kong (Hybrid)
1 Year ago

Get notifed when new similar jobs are uploaded

Jobs in Amsterdam, North Holland, Netherlands

Survay Monkey - Information Security Engineer III

Survay Monkey

Amsterdam, North Holland, Netherlands (Hybrid)
3 Months ago
Sony Interactive Entertainment - Cinematic Animation Director

Sony Interactive Entertainment

Amsterdam, North Holland, Netherlands (On-Site)
3 Weeks ago
miniclip - Game Designer

miniclip

Netherlands (On-Site)
3 Months ago
PlayerUnknown Productions - IT Manager (Part-Time)

PlayerUnknown Productions

Amsterdam, North Holland, Netherlands (Hybrid)
9 Months ago
Team Liquid - Technical Director

Team Liquid

Utrecht, Utrecht, Netherlands (Hybrid)
1 Month ago
Devoteam - Project Leader

Devoteam

Amsterdam, North Holland, Netherlands (On-Site)
1 Month ago
Publicis Groupe - Social Media Content Manager

Publicis Groupe

Amsterdam, North Holland, Netherlands (On-Site)
1 Week ago
IMC - Risk Manager

IMC

Amsterdam, North Holland, Netherlands (On-Site)
1 Week ago
Philips - Consumer Proposition and Insights - Senior Manager

Philips

Amsterdam, North Holland, Netherlands (Hybrid)
1 Week ago
Springer Group - Publisher - Editor-in-Chief Healthcare Management

Springer Group

Utrecht, Utrecht, Netherlands (On-Site)
2 Weeks ago

Get notifed when new similar jobs are uploaded

Data Analysis Jobs

luxsoft - Technical Lead / Senior Data Engineer

luxsoft

Italy, New York, United States (Remote)
1 Month ago
Gameloft - HR Data Analyst / Management Controller

Gameloft

Paris, Île-de-France, France (Hybrid)
2 Weeks ago
binance - Senior QA Engineer - Big Data (Auto & BE Testing)

binance

Taipei City, Taiwan (Hybrid)
1 Year ago
PayPal - Manager, Data Science

PayPal

Dublin, County Dublin, Ireland (Hybrid)
2 Months ago
sitetracker - Senior Business Analyst – Solution Delivery

sitetracker

Austin, Texas, United States (Hybrid)
4 Months ago
Owkin - Data Engineer

Owkin

Paris, Île-de-France, France (On-Site)
1 Week ago
Figma - Software Engineer, Data Infrastructure

Figma

San Francisco, California, United States (Remote)
1 Week ago
Morning Star - Data Research Analyst

Morning Star

Mumbai, Maharashtra, India (On-Site)
2 Weeks ago
Luxoft - Data Engineer for Market Data Projects (with Streamlit Expertise)

Luxoft

Brazil, Indiana, United States (Remote)
8 Months ago
sound cloud - Senior Data Analyst

sound cloud

Berlin, Berlin, Germany (On-Site)
1 Week ago

Get notifed when new similar jobs are uploaded

About The Company

Adyen is a technology company that provides a single platform to accept payments anywhere in the world through any sales channel. Driven by a vision to improve customer experience, streamline processes, and ultimately increase revenue, Adyen enables businesses to process payments across online, mobile, and Point-of-Sale (POS) with over 250 payment methods in 187 transaction currencies. Over 3,500 businesses use the Adyen payment platform, including Facebook, Airbnb, Spotify, Groupon, Evernote, Booking.com, Yelp, Vodafone, Mango, Abercrombie & Fitch, O’Neill, and KLM. Adyen is headquartered in Amsterdam, with offices in San Francisco, São Paulo, Singapore, London, Paris, Berlin, Stockholm, Madrid, and Boston.


Madrid, Community Of Madrid, Spain (Hybrid)

San Francisco, California, United States (On-Site)

Madrid, Community Of Madrid, Spain (On-Site)

New York, United States (On-Site)

San Francisco, California, United States (Hybrid)

Milan, Lombardy, Italy (Hybrid)

New York, United States (On-Site)

Singapore (Hybrid)

San Francisco, California, United States (Hybrid)

Amsterdam, North Holland, Netherlands (Hybrid)

View All Jobs

Get notified when new jobs are added by Adyen

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug