Community

Home >

Jobs >

LEAD DATA ENGINEER

LEAD DATA ENGINEER

1 Month ago • 5 Years + • Data Analyst

About the job

11 skills required for this role

Test your skills to join the top 1% applicants for this job

scala

kubernetes

spark

aws

python

sql

gitlab

tableau

oops

postgresql

mysql

Job Description

As a Lead Data Engineer, you will be responsible for developing an infrastructure for solving product analytics problems based on ClickHouse. You will also build and support data processing pipelines using the Spark on Kubernetes stack, Airflow, Clickhouse, and Python. You will be responsible for prioritizing tasks for a mini-team of data engineers, optimizing infrastructure, helping analysts build complex data processing pipelines, and organizing processes for systematic data collection from different sources. You will be working with a variety of technologies, including Airflow, Spark, Kubernetes, AWS S3, ClickHouse, PostgreSQL, MySQL, Scala, Python, and SQL. You will also be involved in data governance and CI/CD processes.

Must have:

5+ years of experience as a Data Engineer
Experience building ETL on Apache Spark
Understanding of Apache Airflow orchestration
Knowledge of SQL and Python
Experience with AWS S3
Experience with ClickHouse, PostgreSQL, and MySQL
Fluent Russian

Perks:

Relocation to Serbia, Armenia, or other destinations
Relocation program and assistance with paperwork
Competitive salary reviewed every 6 months
Flexible work setup
Flexible benefits: $1000 annual perk allowance
Full pay during sick leaves and personal leaves
20 working days of vacation annually

Not hearing back from companies?

Unlock the secrets to a successful job application and accelerate your journey to your next opportunity.

Our team values an analytical approach to decision making. We accumulate a huge amount of data every day – game events, marketing metrics and much more, which, when carefully analyzed, helps us get valuable insights and improve our games. High quality and speed of processing of this data play a key role for us. And we are looking for an experienced Lead Data Engineer who will help us with this and will be able to take our work with data to the next level.

What we will trust you with

Development of an infrastructure for solving product analytics problems based on ClickHouse.
Building and supporting data processing pipelines on the Spark on Kubernetes stack (Scala and SQL API), Airflow, Clickhouse and Python.
Prioritize tasks for mini-team of data engineers.
Optimization of infrastructure (optimal selection of memory size, cores, disks, instance types) for data processing tasks.
Helping analysts build complex data processing pipelines in Airflow.
Organization of processes for systematic data collection from different sources using different protocols.

Our stack:

Orchestration - AirFlow v2.5.3
ETL - Spark v3.2.0, AWS Glue
Resource Management - Kubernetes
Storage - AWS S3
DB - ClickHouse, PostgreSQL, MySQL
Languages - Scala, Python, SQL
Data governance - DataHub
DQ - Deequ
CI CD - GitLab CI
Secret storage - Vault
BI - Tableau

What we expect from you

Experience as a Data Engineer or similar position for at least five years.
Ability to build ETL on Apache Spark.
Understanding of how to implement orchestration on Apache Airflow.
Knowledge of SQL and the ability to understand Python code.
Experience working with S3.
Experience in working with databases (ClickHouse - must have, PostgreSQL, MySQL) and query optimization.
Fluent Russian.

WHAT WE OFFER

An opportunity to contribute significantly to a globally successful project within a stable international company.
Relocation to Serbia, or Armenia, or another destination upon our mutual agreement.
Relocation program and assistance with paperwork for residence in countries where our offices are located.
Competitive salary, reviewed every 6 months based on market trends and your performance.
Flexible work setup: whether you prefer coming to the office, working from home, or a hybrid of the two, the choice is yours.
Flexible benefits: an annual $1000 to spend on perks, including health insurance, sports, therapy, education and language classes.
Full pay during sick leaves and personal leaves, in addition to 20 working days of vacation annually.

View Full Job Description