Data Engineer – Senior Associate - P&T Labs

Line of Service

Internal Firm Services

Industry/Sector

Not Applicable

Specialism

IFS - Internal Firm Services - Other

Management Level

Senior Associate

Job Description & Summary

A career in Products and Technology is an opportunity to bring PwC's strategy to life by driving products and technology into everything we deliver. Our clients expect us to bring the right people and the right technology to solve their biggest problems; Products and Technology is here to help PwC meet that challenge and accelerate the growth of our business. We have skilled technologists, data scientists, product managers and business strategists who are using technology to accelerate change.

Our team collaborates with product strategy and product managers to govern readiness standards in achieving principles (compliance, privacy, security) by design for what PwC’s technology assets require to be successful in the market. They provide guidance for product development across the lifecycle (ideation / strategy through commercialization / monetization). Additionally, they facilitate market readiness for technology assets overall, as changes occur to assets or market conditions throughout the asset’s life cycle.

Data Engineer (Azure Data Lake, Spark & Databricks)

Required Knowledge and Skills:

6-9 years of experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.

High Proficiency in at least two of Scala, Python, Spark applied to large scale data sets

Expertise with big data technologies (Spark, Data Lake, Delta Lake, Hive)

Knowledge of batch and streaming data processing techniques

Understanding of the Data Lifecycle Management process to collect, access, use, store, transfer, delete data.

Expert level knowledge of using SQL to write complex, highly optimized queries across large volumes of data.

Hands-on object-oriented programming experience using Scala, Python, R, or Java

Proficient with Azure DevOps, managing backlogs and creating/maintaining pipelines

Experience working in an Agile environment and applying it in Azure DevOps

At Effectual, Data engineers are responsible for designing, building, and maintaining datasets that can be leveraged in data projects.

Obsession for service observability, instrumentation, monitoring, and alerting

Knowledge or experience in architectural best practices in building data lakes

Responsibilities

You will be responsible for designing and building optimized data pipelines using emerging technologies, in a cloud environment, for the purpose of driving analytic insights.

Create the conceptual, logical and physical data models.

Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of sources like API, Blob Storage, and no SQL Database

Design, develop, test, deploy, maintain, and improve data integration pipeline for data streaming.

Develop pipeline objects using Apache Spark / Pyspark / Python

Design and develop data pipeline architectures using Databricks, Spark and cloud Services.

Load and performance test data pipelines built using the above-mentioned technologies.

Good to have

Passionate about testing strategy, problem solving, learning new skills, sharing expertise and knowledge.

Always Be Learning

Product / Engineering Mindset

Education (if blank, degree and/or field of study not specified)

Degrees/Field of Study required:

Degrees/Field of Study preferred:

Certifications (if blank, certifications not specified)

Required Skills

Optional Skills

Desired Languages (If blank, desired languages not specified)

Travel Requirements

Not Specified

Available for Work Visa Sponsorship?

No

Government Clearance Required?

No

Job Posting End Date