Jobs

Scholarships

Courses

Companies

Placements

Community

Home >

Jobs >

Pyspark Engineer

New South Wales, Australia (On-site)

Pyspark Engineer

2 Weeks ago • All levels • Data Analyst

About the job

Summary

This role involves designing, developing, and maintaining ETL pipelines using PySpark, optimizing for performance and scalability. The PySpark Engineer will work with large structured and unstructured datasets, transforming data to meet business needs and integrating data from multiple sources. Collaboration with cross-functional teams is key to understanding data requirements and translating them into efficient workflows. Responsibilities include implementing data governance best practices, debugging pipelines, improving performance and reliability, and providing documentation and training. The project focuses on a high-impact data engineering initiative, delivering data-driven insights for business decisions.

12 skills required for this role

Test your skills to join the top 1% applicants for this job

spark

azure

data-visualization

aws

python

sql

tableau

hadoop

power-bi

apache-kafka

scalability

cross-functional

Must have:

Proficiency in PySpark
Strong SQL knowledge
Data Warehousing Concepts
Cloud Platform experience (AWS, GCP, Azure)
Big Data Technologies (Hadoop, Spark)
Data Modeling experience
Strong Python skills

Good to have:

Airflow or other orchestration tools
Apache Kafka knowledge
Data visualization tools (Tableau, Power BI)
Machine learning familiarity
Agile methodology experience
Data governance and compliance knowledge

Not hearing back from companies?

Unlock the secrets to a successful job application and accelerate your journey to your next opportunity.

Project description

We are looking for skilled PySpark Engineers to join our team, working on a high-impact data engineering project. The project involves processing large datasets, optimizing ETL pipelines, and building scalable solutions to manage complex data workflows. The ideal candidate will collaborate closely with data scientists, data analysts, and software engineers to drive robust, data-driven insights for business decisions.

Responsibilities

Design, develop, and maintain ETL pipelines using PySpark, optimizing for performance and scalability.

Work with large volumes of structured and unstructured data, transforming data to meet business needs.

Integrate data from multiple sources into the data platform, ensuring data integrity and quality.

Collaborate with cross-functional teams to understand data requirements and translate them into efficient data workflows.

Implement best practices for data governance, monitoring, and data security.

Debug and troubleshoot issues across ETL pipelines and data workflows.

Continuously improve performance, scalability, and reliability of existing data pipelines.

Provide documentation and training for data workflows and processes.

Skills

Must have

Proficiency in PySpark: In-depth experience with PySpark for data processing and transformation tasks.

SQL Knowledge: Strong command of SQL for querying and processing data.

Data Warehousing Concepts: Familiarity with data warehousing, data lakes, and data integration principles.

Cloud Platforms: Experience with cloud environments like AWS, GCP, or Azure for data storage and processing.

Big Data Technologies: Hands-on experience with Hadoop and Spark ecosystem (Spark SQL, Spark Streaming).

Data Modeling: Experience in designing and implementing efficient data models.

Python Programming: Strong Python skills, particularly in data manipulation and analysis.

Nice to have

Experience with Airflow or Other Orchestration Tools: Knowledge of workflow orchestration tools for scheduling and monitoring data pipelines.

Knowledge of Apache Kafka: Understanding of Kafka for real-time data streaming and integration.

Familiarity with Data Visualization Tools: Knowledge of visualization tools like Tableau, Power BI, or similar.

Machine Learning Exposure: Familiarity with machine learning concepts, particularly with integrating ML models in data workflows.

Agile Methodology: Experience working in Agile/Scrum environments.

Data Governance and Compliance Knowledge: Understanding of data governance frameworks and compliance standards, such as GDPR.

Other

Languages

English: C1 Advanced

Seniority

Senior

View Full Job Description

About The Company

Luxoft

1206 Active Jobs

Luxoft, a DXC Technology Company (NYSE: DXC), is a digital strategy and software engineering firm providing bespoke technology solutions that drive business change for customers the world over. Acquired by U.S. company DXC Technology in 2019, Luxoft is a global operation in 44 cities and 21 countries with an international, agile workforce of nearly 18,000 people. It combines a unique blend of engineering excellence and deep industry expertise, helping over 425 global clients innovate in the areas of automotive, financial services, travel and hospitality, healthcare, life sciences, media and telecommunications.

DXC Technology is a leading Fortune 500 IT services company which helps global companies run their mission critical systems. Together, DXC and Luxoft offer a differentiated customer-value proposition for digital transformation by combining Luxoft’s front-end digital capabilities with DXC’s expertise in IT modernization and integration. Follow our profile for regular updates and insights into technology and business needs.