Senior Data Engineer with Python and Databricks

2 Months ago • All levels • Data Analysis

Job Summary

Job Description

As a Senior Data Engineer, you will collaborate with data scientists and analysts to understand data needs and build efficient data pipelines. Your responsibilities include designing, implementing, and maintaining data pipelines for data ingestion, processing, and transformation using cloud services and other technologies. You will create and maintain data solutions in Azure using services like Data Factory, Synapse, Databricks/Spark, Polars/Pandas, and Fabric. Ensuring data quality, integrity, and dependability by implementing data validation and cleansing procedures is crucial, as is enhancing the scalability, efficiency, and cost-effectiveness of data pipelines. Furthermore, you will be responsible for monitoring and resolving data pipeline issues, ensuring data consistency and availability.
Must have:
  • Create and maintain data pipelines.
  • Implement data validation and cleansing procedures.
  • Work with Azure services like Data Factory, Synapse, and Databricks/Spark.

Job Details

  • Work together with data scientists and analysts to understand the needs for data and create effective data pipelines
  • Solution design, implement, and maintain data pipelines for data ingestion, processing, and transformation using could services and other technologies
  • Create and maintain data solutions IN Azure, using any of the following: Data Factory, Synapse, Databricks/Spark, Polars/Pandas, Fabric 
  • Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
  • Improve the scalability, efficiency, and cost-effectiveness of data pipelines.
  • Monitoring and resolving data pipeline problems, ensuring  consistency and availability of the data.

 

Similar Jobs

plana technologies - Network Engineer

plana technologies

Quito, Pichincha, Ecuador (Remote)
1 Month ago
Trellix - DevOps/Software Engineer

Trellix

Cork, County Cork, Ireland (On-Site)
2 Months ago
PwC - IN_Senior Associate_Azure DevOps Engineer_MS Dynamics_Advisory_Hyderabad

PwC

Hyderabad, Telangana, India (On-Site)
5 Days ago
Veeam Software - Data Analytics Engineer

Veeam Software

(Remote)
3 Months ago
AiDash - Software Development Engineer - II DevOps

AiDash

Bengaluru, Karnataka, India (Hybrid)
8 Months ago
Maliyo Games - Data Analyst

Maliyo Games

Nigeria (On-Site)
8 Months ago
Ambient.ai - Staff Software Engineer - Data Platform

Ambient.ai

Redwood City, California, United States (Hybrid)
8 Months ago
Visa - Lead Data Science

Visa

Almaty, Almaty Region, Kazakhstan (Hybrid)
5 Months ago
Forescout Technologies  Inc  - Senior Data Scientist

Forescout Technologies Inc

Eindhoven, North Brabant, Netherlands (On-Site)
2 Months ago
Luxoft - Senior Data Ops Engineer

Luxoft

(Remote)
8 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

TechVedika - L3 Support/ Infra Cloud Engineer/VM Ware

TechVedika

Hyderabad, Telangana, India (On-Site)
1 Month ago
Varonis  - SOC & IR Specialist

Varonis

United States (Hybrid)
1 Week ago
Celestial AI - SQA Engineer

Celestial AI

Hyderabad, Telangana, India (On-Site)
2 Weeks ago
Trend Micro - Field Sales Engineer

Trend Micro

Raleigh, North Carolina, United States (On-Site)
1 Month ago
Apple - Software Engineering Manager - Apple Services Engineering

Apple

Seattle, Washington, United States (On-Site)
3 Weeks ago
Zones - Software, Cloud and Datacenter Solution Architect

Zones

Bengaluru, Karnataka, India (On-Site)
5 Months ago
Publicis Groupe - Senior Data Analyst

Publicis Groupe

Singapore (Hybrid)
1 Week ago
Blitz app - Senior Data Scientist

Blitz app

Bengaluru, Karnataka, India (Remote)
5 Months ago
Mcdonalds - Full Stack Software Engineer II (C#/.NET)

Mcdonalds

Mexico City, Mexico (On-Site)
3 Weeks ago
Banyan Software - Solution Engineer

Banyan Software

Chennai, Tamil Nadu, India (Hybrid)
6 Days ago

Get notifed when new similar jobs are uploaded

Jobs in Timișoara, Timiș, Romania

Wind River - Senior Member Technical Staff - Linux Kernel

Wind River

Galați, Județul Galați, Romania (On-Site)
1 Month ago
Tesla - Service Manager

Tesla

Timișoara, Timiș, Romania (On-Site)
5 Months ago
Playtika - Senior C# Developer/Tech Lead

Playtika

Romania (Hybrid)
3 Months ago
luxsoft - Senior DevOps Engineer with SRE responsibilities

luxsoft

Bucharest, Bucharest, Romania (Hybrid)
1 Month ago
endava - Senior Database Administrator

endava

Bucharest, Bucharest, Romania (On-Site)
2 Months ago
Playtika - Unity Developer

Playtika

Romania (Hybrid)
7 Months ago
Electronic Arts - Software Engineer, C++ - EA Sports FC

Electronic Arts

Bucharest, Romania (Hybrid)
2 Weeks ago
London stock Exchange - Cloud DevOps Engineer

London stock Exchange

Bucharest, Romania (On-Site)
2 Weeks ago
Electronic Arts - Senior Lighter - EA Sports FC

Electronic Arts

Bucharest, Bucharest, Romania (On-Site)
5 Months ago
Amber - Bazel Senior Build Engineer (Project Based)

Amber

Bucharest, Bucharest, Romania (Remote)
4 Months ago

Get notifed when new similar jobs are uploaded

Data Analysis Jobs

Kavalirio - Managed Care Data Analyst

Kavalirio

Sacramento, California, United States (On-Site)
1 Month ago
dun bradstreet - Senior Director of Data Science, AaaS

dun bradstreet

Frankfurt Am Main, Hessen, Germany (Hybrid)
2 Months ago
Rackspace Technology - Presales Data Science Architect – AWS Cloud

Rackspace Technology

San Antonio, Texas, United States (Remote)
1 Month ago
Sailpoint - Business Systems Analyst (Workday)

Sailpoint

Pune, Maharashtra, India (Hybrid)
2 Months ago
Euromonitor - Data Platform Engineer II

Euromonitor

Bengaluru, Karnataka, India (Hybrid)
3 Months ago
Fortra - Lead Data Scientist

Fortra

Canada (Remote)
3 Weeks ago
Apple - Senior Product Data Scientist, Media Services

Apple

Culver City, California, United States (On-Site)
2 Weeks ago
Nintendo - Sr BI Data Analytics Engineer

Nintendo

Redmond, Washington, United States (On-Site)
3 Months ago
Optiv - Principal Analyst

Optiv

Bengaluru, Karnataka, India (On-Site)
1 Month ago
PwC - Consultant expérimenté Data analyst | CDI | F/H

PwC

Neuilly-sur-Seine, Île-de-France, France (On-Site)
10 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Melbourne, Victoria, Australia (On-Site)

Cluj-Napoca, Cluj County, Romania (On-Site)

Ho Chi Minh City, Vietnam (On-Site)

Singapore (On-Site)

Sydney, New South Wales, Australia (Hybrid)

Brisbane, Queensland, Australia (Hybrid)

Sofia, Sofia City Province, Bulgaria (On-Site)

Bucharest, Romania (Hybrid)

Cluj-Napoca, Cluj County, Romania (On-Site)

Bogotá, Bogota, Colombia (Hybrid)

View All Jobs

Get notified when new jobs are added by endava

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug