Senior Data Engineer

11 Minutes ago • 6 Years +

Data Analysis

Job Description

The Senior Data Engineer will join eBay's Cloud Data Technologies (CDT) team in Dublin, focusing on data infrastructure and the end-to-end data lifecycle. This role involves working with a Hadoop-based data warehouse to build, optimize, and maintain scalable and reliable big data solutions. Key responsibilities include designing and developing robust data pipelines, implementing ETL processes for analytics, optimizing distributed systems, and ensuring data security and integrity. The engineer will collaborate with various teams to align solutions with business needs and contribute to operational excellence.

Must Have:

Design, develop, and maintain robust data pipelines in Hadoop and related ecosystems.
Implement data ETL processes for batch and streaming analytics requirements.
Optimize and troubleshoot distributed systems for ingestion, storage, and processing.
Ensure data security, integrity, and compliance throughout the infrastructure.
Extensive experience with Apache Kafka, Apache Flink, and other relevant streaming technologies.
Proficiency in Hadoop ecosystems such as Hive, Iceberg, Spark sql.
Good Understanding of Apache Airflow tool.
Proven ability to design and implement automated data pipelines and materialized views.
Proficiency in Python, Unix or similar languages.
Good understanding of SQL (Oracle, SQL server or similar languages).
6+ years of directly applicable experience.

Add these skills to join the top 1% applicants for this job

data-analytics

oracle

github

data-structures

game-texts

apache-flink

incident-response

unix

spring-boot

apache-kafka

prometheus

grafana

hadoop

spark

junit

ci-cd

docker

microservices

kubernetes

git

python

algorithms

sql

github-actions

jenkins

java

About the team:

The Cloud Data Technologies (CDT) team at eBay oversees data infrastructure and the management of the end-to-end data lifecycle. As a Data Engineer on our Dublin team, you will work on our Hadoop-based data warehouse, contributing to scalable and reliable big data solutions for analytics and business insights. This is a hands-on role focused on building, optimizing, and maintaining large data pipelines and warehouse infrastructure.

What you will accomplish:

Design, develop, and maintain robust data pipelines in Hadoop and related ecosystems, ensuring data reliability, scalability, and performance.
Implement data ETL processes for batch and streaming analytics requirements.
Optimize and troubleshoot distributed systems for ingestion, storage, and processing.
Collaborate with data engineers, analysts, and platform engineers to align solutions with business needs.
Ensure data security, integrity, and compliance throughout the infrastructure.
Maintain documentation and contribute to architecture reviews.
Participate in incident response and operational excellence initiatives for the data warehouse.
Continuously learn and apply new Hadoop ecosystem tools and data technologies

What you will bring:

Extensive experience with Apache Kafka, Apache Flink, and other relevant streaming technologies.
Proficiency in Hadoop ecosystems such as Hive, Iceberg, Spark sql.
Good Understanding of Apache Airflow tool for orchestrating complex computational workflows and data processing pipelines.
Proven ability to design and implement automated data pipelines and materialized views.
Proficiency in Python, Unix or similar languages.
Good understanding of SQL oracle, SQL server or similar languages.
Java + Spring Boot: Build and maintain microservices.
Ops & CI/CD: Monitoring (Prometheus/Grafana), logging, pipelines (Jenkins/GitHub Actions).
Core Engineering: Data structures/algorithms, testing (JUnit/pytest), Git, clean code.
Cloud Native: Docker, Kubernetes (deploy, network, scale, troubleshoot).
6+ years of directly applicable experience
BS in Computer Science, Engineering, or equivalent experience.