Staff Software Engineer - Spark and OLAP Internals

This is Adyen

Adyen provides payments, data, and financial products in a single solution for customers like Meta, Uber, H&M, and Microsoft - making us the financial technology platform of choice. At Adyen, everything we do is engineered for ambition. 

For our teams, we create an environment with opportunities for our people to succeed, backed by the culture and support to ensure they are enabled to truly own their careers. We are motivated individuals who tackle unique technical challenges at scale and solve them as a team. Together, we deliver innovative and ethical solutions that help businesses achieve their ambitions faster.

Staff Software Engineer - Spark and OLAP Internals

Adyen hosts a significant footprint in both scale and variety of distributed systems. This covers distributed compute (Spark, Trino, Flink), distributed databases (Cassandra, Druid), and distributed file/object storage (HDFS, Ceph Object Gateway). These technologies are offered as-a-service internally, towards product engineering, to support them in building and scaling world-class products.

We’re looking for an expert with deep knowledge in distributed systems, to both improve operations/scalability of existing offerings, as well as introduce and mature new ones. The initial focus will be scaling and tuning our Hadoop and Spark infrastructure, in addition to iterating on our OLAP platform, where we currently use Apache Druid. Over time, we expect this expertise to be useful in a wider range of distributed systems internally.

What you’ll do

You’ll be asked to both build new as-a-service offerings, and improve the existing ones. This role is perfect for you if you're passionate about one or all of the following:

• Design, build, and optimize data clusters to ensure scalability, fault tolerance, and high availability. Covering both batch and streaming workloads.
• Focus on improving the Spark, Hadoop, Kubernetes, Deltalake, and Druid ecosystems internally.
• Evolve these open-source projects internally, with the intention of contributing this code upstream.
• Educate and grow Adyen’s internal knowledge in these topics. Covering both peer platform engineers, and the platform users.

Who you are

Must have experience in:

• Scaling and tuning large deployments of Spark-on-k8s and Spark-on-Hadoop
• Hadoop and the HDFS protocol
• Designing and tuning shuffle heavy systems, on yarn, or on k8s via remote shuffle services
• One of the lakehouse file formats (Delta, Iceberg, Hudi)
• OLAP technologies covering at least one of Clickhouse, Apache Druid, Apache Pinot, or Apache Doris.
• Open-source contributions in one of the must-have technologies, or other common ones (e.g. Kafka, Cassandra, Trino, etc)
• Team player with strong communication skills
• Ability to work closely with diverse stakeholders you enable (analysts, data scientists, data engineers, etc.) and depend upon (infrastructure, security, etc).
• Demonstrated ability to troubleshoot and resolve issues in large-scale, production environments with distributed systems.
  
  Nice-to-have experience in:
• Next generation and multi-modal data formats (e.g. LanceDB)
• Building self-service stateful platforms
• Open-source S3 alternatives (e.g. ceph, minio, etc)
• Native or accelerated runtimes for Spark (Apache DataFusion Comet, Apache Gluten, Nvidia RAPIDS, etc)

Our Diversity, Equity and Inclusion commitments 

Our unique approach is a product of our diverse perspectives. This diversity of backgrounds and cultures is essential in helping us maintain our momentum. Our business and technical challenges are unique, and we need as many different voices as possible to join us in solving them - voices like yours. No matter who you are or where you’re from, we welcome you to be your true self at Adyen. 

Studies show that women and members of underrepresented communities apply for jobs only if they meet 100% of the qualifications. Does this sound like you? If so, Adyen encourages you to reconsider and apply. We look forward to your application!

What’s next?

Ensuring a smooth and enjoyable candidate experience is critical for us. We aim to get back to you regarding your application within 5 business days. Our interview process tends to take about 4 weeks to complete, but may fluctuate depending on the role. Learn more about our hiring process here. Don’t be afraid to let us know if you need more flexibility.

This role is based out of our Amsterdam office. We are an office-first company and value in-person collaboration; we do not offer remote-only roles.