Senior Data Engineer

This role involves designing and maintaining a data solution to gather and transform data from various systems into an Azure Data Lakehouse using Databricks and ADF. Key responsibilities include designing complex ETL processes, building and maintaining data pipelines with Python, improving data quality through anomaly detection, and optimizing data delivery. The Senior Data Engineer will also perform data modeling, collaborate with cross-functional teams, and assist stakeholders with data-related technical issues.

Must Have

• 3-5 years of commercial experience in building and maintaining complex data pipelines
• Proficiency in Python and SQL
• Well-versed in the optimization of ETL processes
• Solid skills in Bash scripting
• Solid knowledge of relational SQL and NoSQL databases (PostgreSQL/Redshift/MongoDB, Snowflake)
• Knowledge of job scheduling and orchestration tools (Airflow etc.)
• Good knowledge of cloud services (Azure)
• Solid expertise in data processing tools for Azure (Elasticsearch, Azure Data Factory, Databricks, Azure Synapse)
• Decent knowledge of CI/CD (Docker, Jenkins, Terraform, Git)
• Good understanding of algorithms and data structures
• Experience in the schema and dimensional data design
• Excellent communication skills, both written and verbal
• English level: upper intermediate+

Good to Have

• Databricks certification
• Microsoft Azure certification

Perks & Benefits

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings
• Other location-specific benefits

Senior Data Engineer (#4290)

REFERRAL BONUS

$1000

Work type: Office/Remote

Technical Level: Senior

Job Category: Software Development

About the project:

Data solution to gather and transform data from multiple heterogeneous systems across our client’s organization into the Azure Data Lakehouse using Databricks and ADF.

Responsibilities:

● Design complex ETL processes of various data sources in the data Lakehouse

● Build new and maintain existing data pipelines using Python to improve efficiency and latency

● Improve data quality through anomaly detection by building and working with internal tools to measure data and automatically detect changes

● Identify, design, and implement internal process improvements, including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes

● Perform data modeling and improve our existing data models for analytics

● Collaborate with SMEs, architects, analysts, data scientists, and others to build solutions that integrate data from many of our enterprise data sources

● Partner with stakeholders, including data, design, product, and executive teams, and assist them with data-related technical issues

Requirements:

● 3-5 years of commercial experience in building and maintaining complex data pipelines

● Proficiency in Python and SQL

● Well-versed in the optimization of ETL processes

● Solid skills in Bash scripting

● Solid knowledge of relational SQL and NoSQL databases: PostgreSQL/Redshift/MongoDB, Snowflake

● Knowledge of job scheduling and orchestration tools (Airflow etc.)

● Good knowledge of cloud services (Azure)

● Solid expertise in data processing tools for Azure: Elasticsearch, Azure Data Factory, Databricks, Azure Synapse

● Decent knowledge of CI/CD (Docker, Jenkins, Terraform, Git)

● Good understanding of algorithms and data structures

● Experience in the schema and dimensional data design

● Excellent communication skills, both written and verbal

● English level: upper intermediate+

Nice to have:

● Databricks certification

● Microsoft Azure certification

We offer*:

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings
• Other location-specific benefits

*not applicable for freelancers