About the job

SummaryBy Outscal

As a Data Engineering Manager, you'll design, develop, and optimize data pipelines using Apache Spark, Flink, Kafka, and Airflow. Experience with cloud platforms like AWS, GCP, or Azure is essential. You'll lead a team of data engineers and oversee data governance, ensuring compliance and accessibility.

About the job

Manager Data Engineering


ABOUT THE JOB Role: Manager Data Engineering Reports to: AD - Data Engineering

Location: Gurgaon

Responsibilities

● As a Data Engineering Manager, you will play a pivotal role in designing the architecture, guiding the development and optimisation of data pipelines and tools. You will mentor the team in building code to efficiently extract raw data, ensuring its quality through robust validation techniques, and leveraging your extensive data expertise to guarantee accurate data ingestion across the pipeline. Your leadership will be instrumental in transforming raw data into formats that are compatible with downstream data sources.

● In this role, you will provide strategic oversight for the development of data tools that are critical for transforming, managing, and accessing data. You will also advise the team on best practices for writing and validating code to ensure the resilience and reliability of data storage platforms.

● You will lead the implementation of performance monitoring protocols across data pipelines, guiding the team in building effective visualisations and aggregations that monitor pipeline health. Additionally, you will coach the team on developing and implementing solutions and self-healing processes that minimize points of failure across various product features.

● Your expertise in data governance will be crucial as you help identify needs and oversee the design of data modelling and handling procedures that ensure compliance with relevant laws and policies. You will also ensure that data accessibility is maintained within your assigned pipelines.

● Collaboration with the Data Science team will be key as you work together on feature set requirements and prediction pipelines. You will prepare team members for meetings with key stakeholders, addressing concerns around data requirements and providing expert guidance on feature estimation. Additionally, you will assist in evaluating data costs, access, usage, dependencies across products, and availability for business or customer scenarios related to one or more product features. Required Knowledge and Skills:

● Data Engineering Tools: Proficiency in ETL/ELT processes, with experience in tools such as Apache Spark, Flink, Kafka, and Airflow.

● Programming Languages: Expertise in programming languages like Python, SQL, and Scala for data processing and analysis.

● Cloud Platforms: Strong experience with cloud-based data services such as AWS, Google Cloud Platform, or Azure.

● Data Warehousing: In-depth knowledge of data warehousing solutions like Snowflake, Redshift-Athena, or BigQuery.

● Visualisation Tools: Experience with data visualisation tools such as Tableau, Power BI, or Looker. ● Pipeline Monitoring: Knowledge of monitoring and alerting tools like Prometheus, Grafana, or Datadog.

● Lake house file formats: Knowledge and experience of working with various storage formats like AVRO, HUDI, parquet, orc and delta.

● API design and development: experience in designing, implementing, and optimising REST APIs, including hands-on expertise in developing robust, scalable, and secure API solutions like flask, Django or fastapi.

Similar Jobs

Unity - Data Engineer

Texas, United States (On-Site)

Electronic Arts - Data Engineer III

Telangana, India (On-Site)

Unity - Data Engineer Manager

California, United States (Hybrid)

Draft Kings - Senior Data Engineer, Snowflake

Massachusetts, United States (On-Site)

PENN Interactive - Senior Data Engineer, Services

Pennsylvania, United States (Hybrid)

PENN Interactive - Senior Data Engineer, Java

Pennsylvania, United States (Hybrid)

PENN Interactive - Data Engineer, Services

Pennsylvania, United States (Hybrid)

PlayStation Global - Marketing Data Engineer Intern - Masters or PhD

California, United States (On-Site)

Skillz - Data Engineer

Karnataka, India (On-Site)

Similar Skill Jobs

Pocket Worlds - User Acquistion Analyst

Texas, United States (On-Site)

Cybersify Cloud computing Pvt ltd - full stack developer

Punjab, India (On-Site)

Easygo - Software Development Engineer, Sportsbook

Victoria, Australia (On-Site)

Streamline Studios - Junior Game Programmer (Unreal)

Federal Territory Of Kuala Lumpur, Malaysia (On-Site)

King - Backend Engineer - Catalog Games

Catalonia, Spain (On-Site)

SEGA - Data Scientist (H/F/NB)

Île-de-France, France (Hybrid)

SEGA - Data Analyst (H/F/NB)

Île-de-France, France (Hybrid)

SEGA - Group HR Systems & Insights Team Lead

England, United Kingdom (Hybrid)

SEGA - Group HR Systems & Insights Specialist

England, United Kingdom (Hybrid)

Jobs in Gurugram, Haryana, India

LogicGo Infotech - 3D Modeler

Gujarat, India (On-Site)

IMAGE Creative Education - 2d Animator

Tamil Nadu, India (On-Site)

Cybersify Cloud computing Pvt ltd - full stack developer

Punjab, India (On-Site)

Huemn - Motion Graphic Designer

Telangana, India (On-Site)

Kaleidoscope - 3D Artist

Tamil Nadu, India (On-Site)

WongDoody - SERVICE DESIGN LEAD

Maharashtra, India (On-Site)

Keywords Studios (Player Support) - DevSecOps Engineer II

Maharashtra, India (On-Site)

DNEG - Head of Legal (India)

Maharashtra, India (On-Site)

Electronic Arts - Data Engineer III

Telangana, India (On-Site)

Longbow Games Pvt. LTD. - Game Producer

Karnataka, India (On-Site)

Software Engineering Jobs

Cybersify Cloud computing Pvt ltd - full stack developer

Punjab, India (On-Site)

Easygo - Software Development Engineer, Sportsbook

Victoria, Australia (On-Site)

Azra Games - Mid-level Software Engineer (Visual Scripter)

California, United States (Hybrid)

Three Space Lab - VR Asset Artist (Contract)

Tennessee, United States (Remote)

PTW - Front of House Executive

England, United Kingdom (Hybrid)

Build A Rocket Boy - Senior Designer (m/f/d)

North Rhine-Westphalia, Germany (On-Site)

King - Backend Engineer - Catalog Games

Catalonia, Spain (On-Site)

Resolution Games - Student Opportunity - Master Thesis

Stockholm County, Sweden (Hybrid)

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug