Senior PySpark Data Engineer

3 Weeks ago • 7 Years + • Data Analyst

About the job

Job Description

Senior PySpark Data Engineer needed for projects in the Middle East. Collaborate with business SMEs to implement data validation rules, understand business requirements and data flow, and work with modern technologies on high-performing platforms. Responsibilities include understanding the data platform lifecycle, contributing to design documents, unit test plans, and code reviews. Requires proficiency in PySpark, Python, SQL, Azure data/delta lake, Azure DevOps, CI/CD, and Git. Experience with data lake houses and business intelligence architecture is essential. Agile experience is a plus.
Must have:
  • 7+ years relevant experience
  • Proficient in PySpark (3+ years)
  • Proficiency in Python & PySpark (5+ years)
  • Advanced SQL knowledge (5+ years)
  • Azure data/delta lake, DevOps, CI/CD, Git
  • Data platform lifecycle understanding
  • Data analysis & understanding complex data
  • Design docs, Unit tests, Code reviews
Good to have:
  • Knowledge of financial markets
  • Portfolio theory and risk management
  • Agile environment experience (Scrum)
Not hearing back from companies?
Unlock the secrets to a successful job application and accelerate your journey to your next opportunity.
Project description

We are hiring for projects in the Middle East. The team comprises front-end and back-end developers, data analysts and data scientists, architects, analysts, and project managers. You will work with modern technologies and be an integral part of continuing to evolve our high-performing platforms. If you are serious about being a part of this fast-paced and exciting journey, this project should be top of the list!
This role will be primarily responsible for closely collaborating with a business SME to implement data validation rules.

Responsibilities

Collaborating with a business SME to implement data validation rules.

Understand business requirements and actively provide input from a data perspective.

Understand the underlying data and flow of data.

Understand overall platform architecture and capabilities and how to load data effectively.

Recognize and adapt to the changes in processes.

Skills

Must have

7 years + of relevant experience.

Proficient hands-on experience with PySpark-based data platforms:

Databricks, Cloudera, Apache Spark, etc. (3y+).

Proficiency in Python and PySpark (5y+).

Advanced knowledge in SQL (5y+).

Should be able to analyse and understand complex data.

Knowledge of Azure data/delta lake, Azure DevOps, CI/CD, and Git is essential.

Should clearly understand the data platform lifecycle and contribute to preparing Design documents, Unit Test plans, and Code review reports.

Knowledge of data Lake house and business intelligence architecture.

Experience working in an Agile environment (Scrum, Agile) is a plus.

Interpersonal/communication skills (both oral/written) with the ability to communicate at various levels with clarity & precision.

Bachelor's degree in IT/Computer Science or related field

Nice to have

Knowledge of the financial markets, portfolio theory, and risk management is a plus

Other

Languages

English: C1 Advanced

Seniority

Senior

View Full Job Description

Add your resume

80%

Upload your resume, increase your shortlisting chances by 80%

About The Company

Luxoft, a DXC Technology Company (NYSE: DXC), is a digital strategy and software engineering firm providing bespoke technology solutions that drive business change for customers the world over. Acquired by U.S. company DXC Technology in 2019, Luxoft is a global operation in 44 cities and 21 countries with an international, agile workforce of nearly 18,000 people. It combines a unique blend of engineering excellence and deep industry expertise, helping over 425 global clients innovate in the areas of automotive, financial services, travel and hospitality, healthcare, life sciences, media and telecommunications.

DXC Technology is a leading Fortune 500 IT services company which helps global companies run their mission critical systems. Together, DXC and Luxoft offer a differentiated customer-value proposition for digital transformation by combining Luxoft’s front-end digital capabilities with DXC’s expertise in IT modernization and integration. Follow our profile for regular updates and insights into technology and business needs.

Poland, Ohio, United States (Remote)

Brazil, Indiana, United States (Remote)

Brazil, Indiana, United States (Remote)

Mexico City, Mexico City, Mexico (Remote)

Ukrainka, Kyiv Oblast, Ukraine (Remote)

Singapore, Singapore (On-Site)

Pune, Maharashtra, India (On-Site)

Poland, Ohio, United States (Remote)

Ukrainka, Kyiv Oblast, Ukraine (Remote)

View All Jobs

Get notified when new jobs are added by Luxoft

Similar Jobs

Cephei Infotech - Senior Full Stack Engineer

Cephei Infotech, India (On-Site)

Paypal - Technical Evangelist

Paypal, United States (Hybrid)

Big Time Studios - Sr. Full-Stack Engineer (South America)

Big Time Studios, (Remote)

ByteDance - Full-Stack Software Engineer - 2025 Start

ByteDance, Singapore (On-Site)

ComeOn Group - Data Engineering Lead

ComeOn Group, Malta (Hybrid)

Head Digital Works - Database Administrator (PostgreSQL)

Head Digital Works, India (On-Site)

Luxoft - Senior/Lead BI Developer

Luxoft, Ukraine (Remote)

Luxoft - Data Analyst

Luxoft, India (On-Site)

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Meta - Software Engineer, Product

Meta, United States (Remote)

Vigaet - Internship-Backend Developer

Vigaet, India (On-Site)

Hacksaw Studios - Gameplay developer / programmer (backend)

Hacksaw Studios, Sweden (On-Site)

Luxoft - Senior Java Developer - Flowable

Luxoft, India (On-Site)

Eightfold - Staff Engineer-Backend

Eightfold, India (Hybrid)

Meta - Software Engineer, Product

Meta, United States (Remote)

Alight Solutions - SWE UI

Alight Solutions, India (Hybrid)

Tranetech Software Solutions - IOS Developer

Tranetech Software Solutions, India (On-Site)

Paypal - Lead Software Engineer (JavaJ2EE)

Paypal, India (Hybrid)

Luxoft - Software Developer

Luxoft, Australia (On-Site)

Get notifed when new similar jobs are uploaded

Jobs in New Delhi, Delhi, India

Anthology  Inc  - Senior Software Engineer II

Anthology Inc , India (On-Site)

Russell Investments - Monitoring Engineer II

Russell Investments, India (On-Site)

DAZN - Content Editor

DAZN, India (On-Site)

Intel Corporation - DFT Engineer

Intel Corporation, India (Hybrid)

DNEG - Project Coordinator

DNEG, India (On-Site)

Saaswika Solutions   - Well Completions Engineer

Saaswika Solutions , India (On-Site)

Abnormal Security - Engineering Manager

Abnormal Security, India (Hybrid)

Bread Financial - Manager - Site Reliability Engineering

Bread Financial, India (Hybrid)

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

Crunchyroll - Principal Data Analyst

Crunchyroll, India (On-Site)

Azra Games - Analytics Lead

Azra Games, United States (Hybrid)

Playtika - Marketing Systems Analyst

Playtika, Israel (On-Site)

AppZen - Data Engineer

AppZen, India (Hybrid)

Patreon - Staff Data Scientist

Patreon, United States (Hybrid)

Luxoft - Senior Business Analyst

Luxoft, United States (Remote)

Next Level Business Services - Data Modeler

Next Level Business Services, United States (On-Site)

Optum - Data Scientist

Optum, India (On-Site)

Get notifed when new similar jobs are uploaded