Senior PySpark Data Engineer
Job Summary
Job Description
- 7+ years relevant experience
- Proficient in PySpark (3+ years)
- Proficient in Python & PySpark (5+ years)
- Advanced SQL knowledge (5+ years)
- Azure data/delta lake, DevOps, CI/CD, Git
- Data platform lifecycle understanding
- Design docs, Unit Tests, Code reviews
- Data Lakehouse & BI architecture knowledge
- Agile experience (Scrum)
- Financial markets, portfolio theory, risk management knowledge
13 skills required
Job Details
Project description
We are hiring for projects in the Middle East. The team comprises front-end and back-end developers, data analysts and data scientists, architects, analysts, and project managers. You will work with modern technologies and be an integral part of continuing to evolve our high-performing platforms. If you are serious about being a part of this fast-paced and exciting journey, this project should be top of the list!
This role will be primarily responsible for closely collaborating with a business SME to implement data validation rules.
Responsibilities
Collaborating with a business SME to implement data validation rules.
Understand business requirements and actively provide input from a data perspective.
Understand the underlying data and flow of data.
Understand overall platform architecture and capabilities and how to load data effectively.
Recognize and adapt to the changes in processes.
Skills
Must have
7 years + of relevant experience.
Proficient hands-on experience with PySpark-based data platforms:
Databricks, Cloudera, Apache Spark, etc. (3y+).
Proficiency in Python and PySpark (5y+).
Advanced knowledge in SQL (5y+).
Should be able to analyse and understand complex data.
Knowledge of Azure data/delta lake, Azure DevOps, CI/CD, and Git is essential.
Should clearly understand the data platform lifecycle and contribute to preparing Design documents, Unit Test plans, and Code review reports.
Knowledge of data Lake house and business intelligence architecture.
Experience working in an Agile environment (Scrum, Agile) is a plus.
Interpersonal/communication skills (both oral/written) with the ability to communicate at various levels with clarity & precision.
Bachelor's degree in IT/Computer Science or related field
Nice to have
Knowledge of the financial markets, portfolio theory, and risk management is a plus
Other
Languages
English: C1 Advanced
Seniority
Senior
Similar Jobs
Unity
ByteDance
Get notifed when new similar jobs are uploaded
Similar Skill Jobs
Salesforce
Scale AI
Get notifed when new similar jobs are uploaded
Jobs in Türkiye
RoofStack
Get notifed when new similar jobs are uploaded
Data Analyst Jobs
PwC
Duolingo
Get notifed when new similar jobs are uploaded
About The Company
Luxoft, a DXC Technology Company (NYSE: DXC), is a digital strategy and software engineering firm providing bespoke technology solutions that drive business change for customers the world over. Acquired by U.S. company DXC Technology in 2019, Luxoft is a global operation in 44 cities and 21 countries with an international, agile workforce of nearly 18,000 people. It combines a unique blend of engineering excellence and deep industry expertise, helping over 425 global clients innovate in the areas of automotive, financial services, travel and hospitality, healthcare, life sciences, media and telecommunications.
DXC Technology is a leading Fortune 500 IT services company which helps global companies run their mission critical systems. Together, DXC and Luxoft offer a differentiated customer-value proposition for digital transformation by combining Luxoft’s front-end digital capabilities with DXC’s expertise in IT modernization and integration. Follow our profile for regular updates and insights into technology and business needs.
Get notified when new jobs are added by Luxoft
