Manager - Data Engineer

6 Months ago • Upto 10 Years • Data Analyst

Job Summary

Job Description

As a Data Engineering Manager, you'll design, develop, and optimize data pipelines using Apache Spark, Flink, Kafka, and Airflow. Experience with cloud platforms like AWS, GCP, or Azure is essential. You'll lead a team of data engineers and oversee data governance, ensuring compliance and accessibility.
Must have:
  • Data Engineering Tools
  • Programming Languages
  • Cloud Platforms
  • Data Warehousing
Good to have:
  • Visualisation Tools
  • Pipeline Monitoring
  • Lake house file formats
  • API design

Job Details

About the job

Manager Data Engineering


ABOUT THE JOB Role: Manager Data Engineering Reports to: AD - Data Engineering

Location: Gurgaon

Responsibilities

● As a Data Engineering Manager, you will play a pivotal role in designing the architecture, guiding the development and optimisation of data pipelines and tools. You will mentor the team in building code to efficiently extract raw data, ensuring its quality through robust validation techniques, and leveraging your extensive data expertise to guarantee accurate data ingestion across the pipeline. Your leadership will be instrumental in transforming raw data into formats that are compatible with downstream data sources.

● In this role, you will provide strategic oversight for the development of data tools that are critical for transforming, managing, and accessing data. You will also advise the team on best practices for writing and validating code to ensure the resilience and reliability of data storage platforms.

● You will lead the implementation of performance monitoring protocols across data pipelines, guiding the team in building effective visualisations and aggregations that monitor pipeline health. Additionally, you will coach the team on developing and implementing solutions and self-healing processes that minimize points of failure across various product features.

● Your expertise in data governance will be crucial as you help identify needs and oversee the design of data modelling and handling procedures that ensure compliance with relevant laws and policies. You will also ensure that data accessibility is maintained within your assigned pipelines.

● Collaboration with the Data Science team will be key as you work together on feature set requirements and prediction pipelines. You will prepare team members for meetings with key stakeholders, addressing concerns around data requirements and providing expert guidance on feature estimation. Additionally, you will assist in evaluating data costs, access, usage, dependencies across products, and availability for business or customer scenarios related to one or more product features. Required Knowledge and Skills:

● Data Engineering Tools: Proficiency in ETL/ELT processes, with experience in tools such as Apache Spark, Flink, Kafka, and Airflow.

● Programming Languages: Expertise in programming languages like Python, SQL, and Scala for data processing and analysis.

● Cloud Platforms: Strong experience with cloud-based data services such as AWS, Google Cloud Platform, or Azure.

● Data Warehousing: In-depth knowledge of data warehousing solutions like Snowflake, Redshift-Athena, or BigQuery.

● Visualisation Tools: Experience with data visualisation tools such as Tableau, Power BI, or Looker. ● Pipeline Monitoring: Knowledge of monitoring and alerting tools like Prometheus, Grafana, or Datadog.

● Lake house file formats: Knowledge and experience of working with various storage formats like AVRO, HUDI, parquet, orc and delta.

● API design and development: experience in designing, implementing, and optimising REST APIs, including hands-on expertise in developing robust, scalable, and secure API solutions like flask, Django or fastapi.

Similar Jobs

Ness Digital - Cloud Data Architect

Ness Digital

Ontario, Canada (Remote)
2 Months ago
Ness Digital - Cloud Data Architect

Ness Digital

United States (Remote)
2 Months ago
The Walt Disney Company - Manager, Software Engineering(Scala)

The Walt Disney Company

New York, New York, United States (On-Site)
5 Months ago
Playrix - Senior Engineering Manager

Playrix

Georgia (Remote)
6 Months ago
Evolution - Scala Spark Developer

Evolution

Riga, Latvia (Hybrid)
11 Months ago
VGW - Risk Intelligence Analyst

VGW

London, England, United Kingdom (Remote)
2 Months ago
Zocdoc - Analyst

Zocdoc

Pune, Maharashtra, India (On-Site)
7 Months ago
Plarium - Business Analytics Lead

Plarium

Herzliya, Tel Aviv District, Israel (On-Site)
2 Months ago
Relax Gaming  - Commercial Analyst

Relax Gaming

Skåne County, Sweden (Hybrid)
4 Months ago
Duolingo - Marketing Technology Manager

Duolingo

New York, New York, United States (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

The Walt Disney Company - Lead Software Engineer, Scala

The Walt Disney Company

Seattle, Washington, United States (On-Site)
3 Months ago
Nielsen Holdings - Big Data Engineers (Java/Scala/Hadoop/Spark/AWS)

Nielsen Holdings

Bengaluru, Karnataka, India (Hybrid)
6 Months ago
The Walt Disney Company - Senior Data Engineer - Identity Data

The Walt Disney Company

New York, New York, United States (On-Site)
4 Months ago
EXUSIA - Senior Data Analyst / Tester

EXUSIA

United States (Remote)
6 Months ago
HiLabs - Data Scientist

HiLabs

Bengaluru, Karnataka, India (On-Site)
5 Months ago
One97 Communications  - Data Science - Lead Data Scientist

One97 Communications

Noida, Uttar Pradesh, India (On-Site)
7 Months ago
Axinous - Staff Software Development Engineer (Backend)

Axinous

Bengaluru, Karnataka, India (On-Site)
4 Months ago
Playrix - Senior Engineering Manager

Playrix

Portugal (Remote)
6 Months ago
ByteDance - Backend Engineer, Ark Large Model Platform - 2025 Start

ByteDance

Singapore (On-Site)
5 Months ago
Epic Games - Technical Director, Machine Learning Programmer

Epic Games

(On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Gurugram, Haryana, India

Ciklum - Expert IOS Developer

Ciklum

Chennai, Tamil Nadu, India (Hybrid)
7 Months ago
Vanderlande - Senior QA Engineer

Vanderlande

Pune, Maharashtra, India (Hybrid)
6 Months ago
Tech Data APAC - Marketing Manager

Tech Data APAC

Maharashtra, India (Hybrid)
6 Months ago
STAGE - Social Media Manager - Haryana

STAGE

Noida, Uttar Pradesh, India (On-Site)
11 Months ago
Dream Sports - MISE - Sales (Delhi)

Dream Sports

Delhi, India (On-Site)
3 Months ago
Craftech 360 - Unity Developer

Craftech 360

Bengaluru, Karnataka, India (On-Site)
9 Months ago
PhonePe - CX Associate Manager, VKYC

PhonePe

Bengaluru, Karnataka, India (On-Site)
5 Months ago
CloudHire - Sitecore Architect/Developer

CloudHire

Mumbai, Maharashtra, India (Remote)
6 Months ago
Paytm - Data Engineering -Senior Data Engineer (Ingest)

Paytm

Noida, Uttar Pradesh, India (On-Site)
5 Months ago
Warner Bros Games - Senior Software Engineer

Warner Bros Games

Hyderabad, Telangana, India (Hybrid)
3 Months ago

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

King - Senior Data Scientist

King

Barcelona, Catalonia, Spain (On-Site)
2 Months ago
The Walt Disney Company - Senior Product Manager II

The Walt Disney Company

Santa Monica, California, United States (On-Site)
3 Months ago
The Walt Disney Company - Senior Data Engineer - Identity Data

The Walt Disney Company

New York, New York, United States (On-Site)
4 Months ago
N-iX - Senior Data Engineer with Snowflake

N-iX

(Hybrid)
3 Months ago
Match Group - Sr. Software Engineer, Data

Match Group

Palo Alto, California, United States (Hybrid)
6 Months ago
Kwalee - Product Analyst

Kwalee

Royal Leamington Spa, England, United Kingdom (On-Site)
3 Months ago
The Walt Disney Company - 2025 Summer Internship – Research Assistant (Consumer Insights)

The Walt Disney Company

Hong Kong (On-Site)
4 Months ago
CloudHire - Database Solution Architect

CloudHire

Gurugram, Haryana, India (Remote)
6 Months ago
PwC - IN_Senior Associate _Visualization Engineer _Data & Analytics _Advisory _Bangalore

PwC

Bengaluru, Karnataka, India (On-Site)
5 Months ago
Social Discovery Group - Chief Data Officer

Social Discovery Group

Belgrade, Serbia (Remote)
2 Months ago

Get notifed when new similar jobs are uploaded