Staff Software Engineer - Spark and OLAP Internals

2 Months ago • All levels • Software Development & Engineering

Job Summary

Job Description

Adyen is seeking a Staff Software Engineer with expertise in distributed systems to improve operations and scalability of existing offerings, as well as introduce and mature new ones. The initial focus will be on scaling and tuning Hadoop and Spark infrastructure, in addition to iterating on their OLAP platform using Apache Druid. The engineer will design, build, and optimize data clusters, improve the Spark, Hadoop, Kubernetes, Deltalake, and Druid ecosystems internally, contribute code upstream to open-source projects, and educate Adyen's internal knowledge in these topics. The role involves collaboration with various stakeholders to troubleshoot and resolve issues in large-scale production environments.
Must have:
  • Scaling and tuning large deployments of Spark-on-k8s and Spark-on-Hadoop
  • Experience with Hadoop and the HDFS protocol
  • Designing and tuning shuffle heavy systems
  • Experience with one of the lakehouse file formats
  • Experience with OLAP technologies
  • Open-source contributions in one of the must-have technologies
  • Strong communication skills and a team player
  • Ability to work with diverse stakeholders
  • Demonstrated ability to troubleshoot and resolve issues
Good to have:
  • Experience with next generation and multi-modal data formats
  • Experience with building self-service stateful platforms
  • Experience with Open-source S3 alternatives
  • Experience with native or accelerated runtimes for Spark

Job Details

This is Adyen

Adyen provides payments, data, and financial products in a single solution for customers like Meta, Uber, H&M, and Microsoft - making us the financial technology platform of choice. At Adyen, everything we do is engineered for ambition. 

For our teams, we create an environment with opportunities for our people to succeed, backed by the culture and support to ensure they are enabled to truly own their careers. We are motivated individuals who tackle unique technical challenges at scale and solve them as a team. Together, we deliver innovative and ethical solutions that help businesses achieve their ambitions faster.

Staff Software Engineer - Spark and OLAP Internals

Adyen hosts a significant footprint in both scale and variety of distributed systems. This covers distributed compute (Spark, Trino, Flink), distributed databases (Cassandra, Druid), and distributed file/object storage (HDFS, Ceph Object Gateway). These technologies are offered as-a-service internally, towards product engineering, to support them in building and scaling world-class products.

We’re looking for an expert with deep knowledge in distributed systems, to both improve operations/scalability of existing offerings, as well as introduce and mature new ones. The initial focus will be scaling and tuning our Hadoop and Spark infrastructure, in addition to iterating on our OLAP platform, where we currently use Apache Druid. Over time, we expect this expertise to be useful in a wider range of distributed systems internally.

What you’ll do

You’ll be asked to both build new as-a-service offerings, and improve the existing ones. This role is perfect for you if you're passionate about one or all of the following:

  • Design, build, and optimize data clusters to ensure scalability, fault tolerance, and high availability. Covering both batch and streaming workloads.
  • Focus on improving the Spark, Hadoop, Kubernetes, Deltalake, and Druid ecosystems internally.
  • Evolve these open-source projects internally, with the intention of contributing this code upstream.
  • Educate and grow Adyen’s internal knowledge in these topics. Covering both peer platform engineers, and the platform users.

Who you are

Must have experience in:

  • Scaling and tuning large deployments of Spark-on-k8s and Spark-on-Hadoop
  • Hadoop and the HDFS protocol
  • Designing and tuning shuffle heavy systems, on yarn, or on k8s via remote shuffle services
  • One of the lakehouse file formats (Delta, Iceberg, Hudi)
  • OLAP technologies covering at least one of Clickhouse, Apache Druid, Apache Pinot, or Apache Doris.
  • Open-source contributions in one of the must-have technologies, or other common ones (e.g. Kafka, Cassandra, Trino, etc)
  • Team player with strong communication skills
  • Ability to work closely with diverse stakeholders you enable (analysts, data scientists, data engineers, etc.) and depend upon (infrastructure, security, etc).
  • Demonstrated ability to troubleshoot and resolve issues in large-scale, production environments with distributed systems.

    Nice-to-have experience in:
  • Next generation and multi-modal data formats (e.g. LanceDB)
  • Building self-service stateful platforms
  • Open-source S3 alternatives (e.g. ceph, minio, etc)
  • Native or accelerated runtimes for Spark (Apache DataFusion Comet, Apache Gluten, Nvidia RAPIDS, etc)

Our Diversity, Equity and Inclusion commitments 

Our unique approach is a product of our diverse perspectives. This diversity of backgrounds and cultures is essential in helping us maintain our momentum. Our business and technical challenges are unique, and we need as many different voices as possible to join us in solving them - voices like yours. No matter who you are or where you’re from, we welcome you to be your true self at Adyen. 

Studies show that women and members of underrepresented communities apply for jobs only if they meet 100% of the qualifications. Does this sound like you? If so, Adyen encourages you to reconsider and apply. We look forward to your application!

What’s next?

Ensuring a smooth and enjoyable candidate experience is critical for us. We aim to get back to you regarding your application within 5 business days. Our interview process tends to take about 4 weeks to complete, but may fluctuate depending on the role. Learn more about our hiring process here. Don’t be afraid to let us know if you need more flexibility.

This role is based out of our Amsterdam office. We are an office-first company and value in-person collaboration; we do not offer remote-only roles.

Similar Jobs

Romero games - Lighting Artist

Romero games

Galway, County Galway, Ireland (Remote)
5 Months ago
Welltech - User Acquisition Manager (Paid Social)

Welltech

Poland (Remote)
3 Weeks ago
Juego Studios - SEO Specialist - North America Region

Juego Studios

Bengaluru, Karnataka, India (On-Site)
8 Months ago
Shield AI - Senior Engineer, Software Autonomy Applications (R3694)

Shield AI

Washington, District Of Columbia, United States (On-Site)
1 Week ago
Sierra - Product Manager, Agent Development

Sierra

New York, United States (On-Site)
3 Months ago
Qualcomm - IT Engineer

Qualcomm

Hyderabad, Telangana, India (On-Site)
2 Months ago
Capgemini - SAP SuccessFactors Administration

Capgemini

Mumbai, Maharashtra, India (On-Site)
2 Months ago
Ion - Lead Software Engineer, Italy

Ion

Milan, Lombardy, Italy (On-Site)
9 Months ago
Qualcomm - Wireless connectivity -Sr Staff Engineer

Qualcomm

Chennai, Tamil Nadu, India (On-Site)
1 Week ago
Trellix - Senior Software Engineer

Trellix

Aylesbury, England, United Kingdom (Hybrid)
3 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Saviynt - Software Architect - Privilege Access Management

Saviynt

United States (Remote)
9 Months ago
Thousand Eyes - Technical Account Manager - Federal

Thousand Eyes

(Remote)
4 Months ago
oni - Business Development Associate

oni

Boston, Massachusetts, United States (Remote)
3 Months ago
PlaySimple - Senior Game Producer/Game Producer

PlaySimple

(On-Site)
5 Months ago
Aristocrat - Business Analyst

Aristocrat

Las Vegas, Nevada, United States (Hybrid)
2 Months ago
WebFX - Software Engineer

WebFX

Ann Arbor, Michigan, United States (On-Site)
8 Months ago
The Walt Disney Company - Electronic Technician (Sound Mechanic)

The Walt Disney Company

Anaheim, California, United States (On-Site)
3 Months ago
Synechron - Senior Java Developer (Angular and Oracle)

Synechron

Hyderabad, Telangana, India (On-Site)
2 Weeks ago
The E.W. Scripps Company - Senior Reporter

The E.W. Scripps Company

Colorado Springs, Colorado, United States (On-Site)
1 Year ago
Univision - Senior Manager, Asi Studios

Univision

New York, United States (On-Site)
1 Week ago

Get notifed when new similar jobs are uploaded

Jobs in Amsterdam, North Holland, Netherlands

grendel games - Technical art intern

grendel games

Leeuwarden, Friesland, Netherlands (Hybrid)
2 Months ago
miniclip - Game Designer

miniclip

Netherlands (On-Site)
3 Months ago
fluence - Logistics Specialist

fluence

Amsterdam, North Holland, Netherlands (Hybrid)
1 Month ago
Tesla - Aftersales Support Specialist

Tesla

North Holland, Netherlands (On-Site)
5 Months ago
Game drive  - Influencer Manager

Game drive

Netherlands (On-Site)
4 Weeks ago
Adyen - Staff Software Engineer - Distributed Data Systems

Adyen

Amsterdam, North Holland, Netherlands (On-Site)
2 Months ago
Beyond Sports - System Admin/IT Support

Beyond Sports

Alkmaar, North Holland, Netherlands (On-Site)
4 Months ago
Square - Senior Azure DevOps Engineer

Square

Groningen, Groningen, Netherlands (Hybrid)
1 Week ago
IMC - Data Quality Engineer

IMC

Amsterdam, North Holland, Netherlands (On-Site)
4 Weeks ago
Tesla - Service Technician

Tesla

Roermond, Limburg, Netherlands (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Software Development & Engineering Jobs

Vercel - Software Engineer, Accounts

Vercel

Berlin, Berlin, Germany (Hybrid)
4 Weeks ago
flix interactive - Senior Engineer

flix interactive

United Kingdom (Remote)
2 Months ago
extreme network - Manager of Systems Engineering - Western United States

extreme network

California, United States (Remote)
2 Months ago
Kaedim - Software Engineer

Kaedim

San Francisco, California, United States (On-Site)
1 Year ago
smarsh - Manager, Engineering

smarsh

Belfast, Northern Ireland, United Kingdom (Remote)
2 Weeks ago
Unity - Staff Software Engineer

Unity

San Francisco, California, United States (Hybrid)
3 Months ago
brandovilla - Senior Software Engineer

brandovilla

San Diego, California, United States (Remote)
11 Months ago
Shield AI - Propulsion Engineer II

Shield AI

Dallas, Texas, United States (On-Site)
1 Week ago
Uniswap Labs - Engineering Manager- Protocol

Uniswap Labs

New York, United States (Remote)
3 Weeks ago
broadcom - Wafer Fab Engineer

broadcom

Breinigsville, Pennsylvania, United States (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Adyen is a technology company that provides a single platform to accept payments anywhere in the world through any sales channel. Driven by a vision to improve customer experience, streamline processes, and ultimately increase revenue, Adyen enables businesses to process payments across online, mobile, and Point-of-Sale (POS) with over 250 payment methods in 187 transaction currencies. Over 3,500 businesses use the Adyen payment platform, including Facebook, Airbnb, Spotify, Groupon, Evernote, Booking.com, Yelp, Vodafone, Mango, Abercrombie & Fitch, O’Neill, and KLM. Adyen is headquartered in Amsterdam, with offices in San Francisco, São Paulo, Singapore, London, Paris, Berlin, Stockholm, Madrid, and Boston.


Madrid, Community Of Madrid, Spain (Hybrid)

San Francisco, California, United States (On-Site)

Madrid, Community Of Madrid, Spain (On-Site)

New York, United States (On-Site)

San Francisco, California, United States (Hybrid)

Milan, Lombardy, Italy (Hybrid)

New York, United States (On-Site)

Singapore (Hybrid)

San Francisco, California, United States (Hybrid)

Amsterdam, North Holland, Netherlands (Hybrid)

View All Jobs

Get notified when new jobs are added by Adyen

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug