Staff Software Engineer - Spark and OLAP Internals

3 Hours ago • All levels

Job Summary

Job Description

Adyen is seeking a Staff Software Engineer with expertise in distributed systems to improve operations and scalability of existing offerings, as well as introduce and mature new ones. The initial focus will be on scaling and tuning Hadoop and Spark infrastructure, in addition to iterating on their OLAP platform using Apache Druid. The engineer will design, build, and optimize data clusters, improve the Spark, Hadoop, Kubernetes, Deltalake, and Druid ecosystems internally, contribute code upstream to open-source projects, and educate Adyen's internal knowledge in these topics. The role involves collaboration with various stakeholders to troubleshoot and resolve issues in large-scale production environments.
Must have:
  • Scaling and tuning large deployments of Spark-on-k8s and Spark-on-Hadoop
  • Experience with Hadoop and the HDFS protocol
  • Designing and tuning shuffle heavy systems
  • Experience with one of the lakehouse file formats
  • Experience with OLAP technologies
  • Open-source contributions in one of the must-have technologies
  • Strong communication skills and a team player
  • Ability to work with diverse stakeholders
  • Demonstrated ability to troubleshoot and resolve issues
Good to have:
  • Experience with next generation and multi-modal data formats
  • Experience with building self-service stateful platforms
  • Experience with Open-source S3 alternatives
  • Experience with native or accelerated runtimes for Spark

Job Details

This is Adyen

Adyen provides payments, data, and financial products in a single solution for customers like Meta, Uber, H&M, and Microsoft - making us the financial technology platform of choice. At Adyen, everything we do is engineered for ambition. 

For our teams, we create an environment with opportunities for our people to succeed, backed by the culture and support to ensure they are enabled to truly own their careers. We are motivated individuals who tackle unique technical challenges at scale and solve them as a team. Together, we deliver innovative and ethical solutions that help businesses achieve their ambitions faster.

Staff Software Engineer - Spark and OLAP Internals

Adyen hosts a significant footprint in both scale and variety of distributed systems. This covers distributed compute (Spark, Trino, Flink), distributed databases (Cassandra, Druid), and distributed file/object storage (HDFS, Ceph Object Gateway). These technologies are offered as-a-service internally, towards product engineering, to support them in building and scaling world-class products.

We’re looking for an expert with deep knowledge in distributed systems, to both improve operations/scalability of existing offerings, as well as introduce and mature new ones. The initial focus will be scaling and tuning our Hadoop and Spark infrastructure, in addition to iterating on our OLAP platform, where we currently use Apache Druid. Over time, we expect this expertise to be useful in a wider range of distributed systems internally.

What you’ll do

You’ll be asked to both build new as-a-service offerings, and improve the existing ones. This role is perfect for you if you're passionate about one or all of the following:

  • Design, build, and optimize data clusters to ensure scalability, fault tolerance, and high availability. Covering both batch and streaming workloads.
  • Focus on improving the Spark, Hadoop, Kubernetes, Deltalake, and Druid ecosystems internally.
  • Evolve these open-source projects internally, with the intention of contributing this code upstream.
  • Educate and grow Adyen’s internal knowledge in these topics. Covering both peer platform engineers, and the platform users.

Who you are

Must have experience in:

  • Scaling and tuning large deployments of Spark-on-k8s and Spark-on-Hadoop
  • Hadoop and the HDFS protocol
  • Designing and tuning shuffle heavy systems, on yarn, or on k8s via remote shuffle services
  • One of the lakehouse file formats (Delta, Iceberg, Hudi)
  • OLAP technologies covering at least one of Clickhouse, Apache Druid, Apache Pinot, or Apache Doris.
  • Open-source contributions in one of the must-have technologies, or other common ones (e.g. Kafka, Cassandra, Trino, etc)
  • Team player with strong communication skills
  • Ability to work closely with diverse stakeholders you enable (analysts, data scientists, data engineers, etc.) and depend upon (infrastructure, security, etc).
  • Demonstrated ability to troubleshoot and resolve issues in large-scale, production environments with distributed systems.

    Nice-to-have experience in:
  • Next generation and multi-modal data formats (e.g. LanceDB)
  • Building self-service stateful platforms
  • Open-source S3 alternatives (e.g. ceph, minio, etc)
  • Native or accelerated runtimes for Spark (Apache DataFusion Comet, Apache Gluten, Nvidia RAPIDS, etc)

Our Diversity, Equity and Inclusion commitments 

Our unique approach is a product of our diverse perspectives. This diversity of backgrounds and cultures is essential in helping us maintain our momentum. Our business and technical challenges are unique, and we need as many different voices as possible to join us in solving them - voices like yours. No matter who you are or where you’re from, we welcome you to be your true self at Adyen. 

Studies show that women and members of underrepresented communities apply for jobs only if they meet 100% of the qualifications. Does this sound like you? If so, Adyen encourages you to reconsider and apply. We look forward to your application!

What’s next?

Ensuring a smooth and enjoyable candidate experience is critical for us. We aim to get back to you regarding your application within 5 business days. Our interview process tends to take about 4 weeks to complete, but may fluctuate depending on the role. Learn more about our hiring process here. Don’t be afraid to let us know if you need more flexibility.

This role is based out of our Amsterdam office. We are an office-first company and value in-person collaboration; we do not offer remote-only roles.

Similar Jobs

Infosys - Java FullStack Developer

Infosys

Mexico City, Mexico City, Mexico (Hybrid)
2 Weeks ago
Visa - Staff Data Engineer

Visa

Warsaw, Masovian Voivodeship, Poland (Hybrid)
6 Months ago
Dream Sports - SDE 2 - React Native

Dream Sports

Mumbai, Maharashtra, India (On-Site)
6 Months ago
Shyft Labs - Data Lead

Shyft Labs

Noida, Uttar Pradesh, India (On-Site)
1 Month ago
Reddit - Senior Software Engineer, Security Partner

Reddit

New York, New York, United States (On-Site)
4 Days ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Velotio Technologies - Lead Engineer (Java)

Velotio Technologies

Pune, Maharashtra, India (Remote)
1 Month ago
NCR Voyix - Scrum Master

NCR Voyix

Hyderabad, Telangana, India (On-Site)
1 Year ago
Capgemini - SAP Finance Solution Expert

Capgemini

Bengaluru, Karnataka, India (On-Site)
1 Week ago
Dream Sports - SDE 3 - Backend (Platform)

Dream Sports

Mumbai, Maharashtra, India (On-Site)
1 Month ago
Nium - Data Engineer II

Nium

Hyderabad, Telangana, India (On-Site)
2 Months ago
Acceldata - Senior SDET - UI & Database

Acceldata

Bengaluru, Karnataka, India (On-Site)
11 Months ago
balbex - Sr Staff/Principal Devops Engineer

balbex

Gurugram, India (On-Site)
9 Months ago
Accurate - Data Architect

Accurate

Hyderabad, Telangana, India (Hybrid)
2 Weeks ago
Warner Bros Games - Staff Data Engineer

Warner Bros Games

Atlanta, Georgia, United States (Hybrid)
2 Months ago
Reddit - Senior Software Engineer, Security Partner

Reddit

New York, New York, United States (On-Site)
4 Days ago

Get notifed when new similar jobs are uploaded

Jobs in Amsterdam, North Holland, Netherlands

Nium - SVP - Business Development EMEA

Nium

Amsterdam, North Holland, Netherlands (Hybrid)
1 Week ago
Progress carrers - Associate Renewals Account Manager

Progress carrers

Rotterdam, South Holland, Netherlands (Hybrid)
1 Month ago
Tesla - Contract Compliance Analyst

Tesla

North Holland, Netherlands (On-Site)
3 Months ago
Tesla - Senior Software Engineer (Backend, .Net)

Tesla

North Holland, Netherlands (On-Site)
3 Months ago
Paradox Interactive - 3D Character Art Internship

Paradox Interactive

Delft, South Holland, Netherlands (On-Site)
1 Month ago
IMC - Quantitative Developer

IMC

Amsterdam, North Holland, Netherlands (On-Site)
1 Week ago
Adyen - Data Engineer

Adyen

Amsterdam, North Holland, Netherlands (On-Site)
2 Weeks ago
Tesla - Senior Legal Counsel, Benelux

Tesla

Amsterdam, North Holland, Netherlands (On-Site)
3 Months ago
Google - Facilities Technician, Electrical

Google

Eemshaven, Groningen, Netherlands (On-Site)
1 Month ago
Marsh McLennan - Business Developer

Marsh McLennan

Rotterdam, South Holland, Netherlands (Hybrid)
1 Week ago

Get notifed when new similar jobs are uploaded

Similar Category Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

About The Company

Adyen is a technology company that provides a single platform to accept payments anywhere in the world through any sales channel. Driven by a vision to improve customer experience, streamline processes, and ultimately increase revenue, Adyen enables businesses to process payments across online, mobile, and Point-of-Sale (POS) with over 250 payment methods in 187 transaction currencies. Over 3,500 businesses use the Adyen payment platform, including Facebook, Airbnb, Spotify, Groupon, Evernote, Booking.com, Yelp, Vodafone, Mango, Abercrombie & Fitch, O’Neill, and KLM. Adyen is headquartered in Amsterdam, with offices in San Francisco, São Paulo, Singapore, London, Paris, Berlin, Stockholm, Madrid, and Boston.


Madrid, Community Of Madrid, Spain (Hybrid)

Bengaluru, Karnataka, India (On-Site)

Amsterdam, North Holland, Netherlands (On-Site)

Madrid, Community Of Madrid, Spain (On-Site)

Amsterdam, North Holland, Netherlands (On-Site)

Amsterdam, North Holland, Netherlands (On-Site)

Amsterdam, North Holland, Netherlands (On-Site)

San Francisco, California, United States (On-Site)

Amsterdam, North Holland, Netherlands (Hybrid)

View All Jobs

Get notified when new jobs are added by Adyen

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug