Associate Staff Engineer - Data Engineering

7 Months ago • All levels

Devops

Job Description

As an Associate Staff Engineer - Data Engineering, you'll design, develop, and manage workflows using Apache Airflow, ensuring efficient and reliable scheduled DAGs. You'll leverage Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Databricks, PySpark, and Python for data integration, storage, pipeline development, and ETL processes. Responsibilities include monitoring, troubleshooting, and optimizing data workflows for high availability and performance. Collaboration with cross-functional teams is key, along with managing virtual machine environments via terminal commands. This role requires strong expertise in Azure cloud services and data engineering principles.

Good To Have:

Other Azure services
DevOps practices
CI/CD in data engineering
SQL and NoSQL databases
Data security and compliance

Must Have:

Azure Data Factory
Azure Pipeline Management
Databricks
Apache Airflow
Python
PySpark
Data pipeline optimization
VM terminal operations

Add these skills to join the top 1% applicants for this job

ci-cd

azure

nosql

python

sql

data-analytics

cross-functional

problem-solving

Company Description

We are a Digital Product Engineering company that is scaling in a big way! We build products, services, and experiences that inspire, excite, and delight. We work at scale — across all devices and digital mediums, and our people exist everywhere in the world (19000+ experts across 33 countries, to be exact). Our work culture is dynamic and non-hierarchical. We are looking for great new colleagues. That is where you come in!

Job Description

We are seeking a skilled and experienced Data Engineer who can assist with managing scheduled DAGs and data workflows. The ideal candidate should have expertise in modern data engineering tools and technologies, particularly in the Azure ecosystem, and be comfortable working in a virtual machine environment via terminal.

Key Responsibilities:

Design, develop, and manage workflows using Apache Airflow, ensuring scheduled DAGs run efficiently and reliably.
Work with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS) for data integration and storage solutions.
Develop and maintain data pipelines and ETL processes using Databricks, PySpark, and Python.
Monitor, troubleshoot, and optimize data workflows and pipelines to ensure high availability and performance.
Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions. Manage and operate within virtual machine environments using terminal commands for development and debugging.

Qualifications

Must have Skills: Azure Data Factory, Azure Pipeline Management (Strong), Databricks, Apache Airflow, Python (strong).

Proficiency in Apache Airflow for orchestrating workflows and managing DAGs.
Strong experience with Azure cloud services, including ADF and ADLS.
Hands-on experience with Databricks and PySpark for big data processing and analytics.
Advanced programming skills in Python, with a focus on data manipulation and automation.
Comfort with terminal operations and commands in VM environments.
Solid understanding of data engineering principles, including data modeling, ETL processes, and data pipeline optimization.
Strong problem-solving skills and the ability to work independently in a dynamic environment.
Experience with other Azure services or similar cloud platforms is a plus.
Familiarity with DevOps practices and tools for CI/CD in data engineering is a plus.
Knowledge of database technologies such as SQL and NoSQL systems is a plus.
Understanding of data security and compliance standards is a plus.