Associate Staff Engineer - Data Engineering

15 Hours ago • All levels • DevOps • Data Analyst

Job Summary

Job Description

As an Associate Staff Engineer - Data Engineering, you'll design, develop, and manage workflows using Apache Airflow, ensuring efficient and reliable scheduled DAGs. You'll leverage Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Databricks, PySpark, and Python for data integration, storage, pipeline development, and ETL processes. Responsibilities include monitoring, troubleshooting, and optimizing data workflows for high availability and performance. Collaboration with cross-functional teams is key, along with managing virtual machine environments via terminal commands. This role requires strong expertise in Azure cloud services and data engineering principles.
Must have:
  • Azure Data Factory
  • Azure Pipeline Management
  • Databricks
  • Apache Airflow
  • Python
  • PySpark
  • Data pipeline optimization
  • VM terminal operations
Good to have:
  • Other Azure services
  • DevOps practices
  • CI/CD in data engineering
  • SQL and NoSQL databases
  • Data security and compliance

Job Details

Company Description

We are a Digital Product Engineering company that is scaling in a big way! We build products, services, and experiences that inspire, excite, and delight. We work at scale — across all devices and digital mediums, and our people exist everywhere in the world (19000+ experts across 33 countries, to be exact). Our work culture is dynamic and non-hierarchical. We are looking for great new colleagues. That is where you come in!

Job Description

We are seeking a skilled and experienced Data Engineer who can assist with managing scheduled DAGs and data workflows. The ideal candidate should have expertise in modern data engineering tools and technologies, particularly in the Azure ecosystem, and be comfortable working in a virtual machine environment via terminal.

Key Responsibilities:

  • Design, develop, and manage workflows using Apache Airflow, ensuring scheduled DAGs run efficiently and reliably.
  • Work with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS) for data integration and storage solutions.
  • Develop and maintain data pipelines and ETL processes using Databricks, PySpark, and Python.
  • Monitor, troubleshoot, and optimize data workflows and pipelines to ensure high availability and performance.
  • Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions. Manage and operate within virtual machine environments using terminal commands for development and debugging.

Qualifications

Must have Skills: Azure Data Factory, Azure Pipeline Management (Strong), Databricks, Apache Airflow, Python (strong).

  • Proficiency in Apache Airflow for orchestrating workflows and managing DAGs.
  • Strong experience with Azure cloud services, including ADF and ADLS.
  • Hands-on experience with Databricks and PySpark for big data processing and analytics.
  • Advanced programming skills in Python, with a focus on data manipulation and automation.
  • Comfort with terminal operations and commands in VM environments.
  • Solid understanding of data engineering principles, including data modeling, ETL processes, and data pipeline optimization.
  • Strong problem-solving skills and the ability to work independently in a dynamic environment.
  • Experience with other Azure services or similar cloud platforms is a plus.
  • Familiarity with DevOps practices and tools for CI/CD in data engineering is a plus.
  • Knowledge of database technologies such as SQL and NoSQL systems is a plus.
  • Understanding of data security and compliance standards is a plus.

Similar Jobs

Playtika - Mobile/C# Developer

Playtika

Romania (Hybrid)
3 Months ago
Lost Boys Interactive - DevOps - GoLang Developer

Lost Boys Interactive

(Remote)
1 Month ago
CloudLinux - SDET / QA Automation Engineer

CloudLinux

Sofia City Province, Bulgaria (Remote)
1 Day ago
Krafton  - [Publishing Platform Div.] Sr. Web Back-End Developer (5년 이상)

Krafton

Seoul, South Korea (On-Site)
3 Months ago
Truecaller - Data Engineer

Truecaller

Stockholm, Stockholm County, Sweden (On-Site)
3 Months ago
VGW - Staff Site Reliability Engineer

VGW

Perth, Western Australia, Australia (On-Site)
1 Week ago
Brillio - Azure DB Architect - Migration - R01531206

Brillio

Bengaluru, Karnataka, India (Hybrid)
4 Months ago
DEVOTEAM - Distributed Cloud | Multicloud FinOps

DEVOTEAM

Lisbon, Lisbon, Portugal (Remote)
4 Months ago
Microsoft - Senior Hardware Engineer

Microsoft

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Microsoft - Site Reliability Engineer

Microsoft

Bucharest, Bucharest, Romania (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

ION - Cloud Engineer/Architect (DevOps)

ION

Italy (On-Site)
4 Months ago
ION - Senior Security Architect

ION

Italy (On-Site)
4 Months ago
Ubisoft - Lead Programmer

Ubisoft

Pune, Maharashtra, India (On-Site)
4 Days ago
Like Card - Senior AI Solutions Architect

Like Card

Amman, Amman Governorate, Jordan (On-Site)
3 Days ago
Nielsen Holdings - Senior Software Engineer - Java, AWS

Nielsen Holdings

Mumbai, Maharashtra, India (Hybrid)
1 Month ago
PwC - IN_Senior Associate_AWS Devops Engineer_Data &Analytics_Advisory_Bangalore

PwC

Bengaluru, Karnataka, India (On-Site)
4 Months ago
Ubisoft - DevOps Linux Administrator

Ubisoft

Saint-Mandé, Île-de-France, France (Hybrid)
2 Weeks ago
Warner Bros Games - Software Engineer II - Backend (Adtech Team)

Warner Bros Games

Bengaluru, Karnataka, India (Hybrid)
2 Weeks ago
Omnissa - Member of technical staff (C++, JavaScript, React, HTML5)

Omnissa

Bengaluru, Karnataka, India (Hybrid)
3 Months ago
Gunzilla - DevOps/Build Engineer

Gunzilla

Kyiv, Kyiv City, Ukraine (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in Colombia

Coupa Software - Pre Sales Technical Consultant

Coupa Software

Colombia (On-Site)
4 Months ago
Neostella - Legal Case Coordinator

Neostella

Medellín, Antioquia, Colombia (On-Site)
3 Months ago
Neostella - Intake Specialist (Weekend Hours Required)

Neostella

Colombia (On-Site)
2 Weeks ago
Evolution - Account Payables Specialist

Evolution

Medellín, Antioquia, Colombia (On-Site)
1 Month ago
Unity - Digital Customer Success Specialist - Data Focus

Unity

Pereira, Risaralda, Colombia (On-Site)
4 Months ago
N-iX - Junior Automation QA Engineer (Python)

N-iX

Colombia (Remote)
2 Days ago
Nagarro - Associate Staff Engineer - Technical Project Manager

Nagarro

Colombia (Remote)
1 Week ago
N-iX - Senior Java Engineer

N-iX

Colombia (Remote)
3 Weeks ago
N-iX - Senior .NET Full Stack Engineer

N-iX

Colombia (Remote)
2 Weeks ago
Anthology  Inc  - Senior Scrum Master

Anthology Inc

Bogotá, Bogota, Colombia (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

DevOps Jobs

Ubisoft - Fullstack Developer Assistant

Ubisoft

Bordeaux, Nouvelle-Aquitaine, France (On-Site)
1 Week ago
Anthology  Inc  - Associate Software Engineer II

Anthology Inc

Brno, South Moravian Region, Czechia (On-Site)
3 Weeks ago
Luxoft - Senior Python Developer with Networking

Luxoft

(Remote)
3 Months ago
NVIDIA - Software Platform Support Engineer - GPU Cloud

NVIDIA

United States (Remote)
2 Weeks ago
Microsoft - ROP - Senior Software Engineer

Microsoft

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Xsolla - VP of Infrastructure and Operations

Xsolla

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (Hybrid)
3 Months ago
PwC - ETIC, GCP/Oracle Cloud Engineer - Manager

PwC

Cairo, Cairo Governorate, Egypt (On-Site)
4 Months ago
Microsoft - Technical Support Engineer

Microsoft

Bengaluru, Karnataka, India (Hybrid)
1 Month ago
PwC - D365 Azure Integration Developer-Manager

PwC

Kolkata, West Bengal, India (On-Site)
5 Months ago
Rackspace Technology - AWS Support Engineer I

Rackspace Technology

Gurugram, Haryana, India (Remote)
2 Weeks ago

Get notifed when new similar jobs are uploaded