Databricks Lead Engineer | Senior Manager | Data & AI | Advisory

At PwC, our data and analytics team leverages data to drive insights and make informed business decisions, using advanced analytics to help clients optimize operations. This role focuses on utilizing advanced analytical techniques to extract insights from large datasets, supporting data-driven decision-making with skills in data manipulation, visualization, and statistical modeling. As a Lead Developer on the Data & AI team, you will spearhead the development and delivery of analytics projects, focusing on Databricks technology for complex healthcare systems. Your role involves leading, reviewing, and managing code, ensuring high standards of development, data quality, compliance, and reliability, contributing to robust data solutions and improved decision-making for healthcare.

Must Have

• Develop scalable data solutions, define data models, design data lakes/lakehouses, and create high-level design documents.
• Build and optimize complex data pipelines (batch and streaming) using PySpark/Scala, Delta Lake, and SQL, automating workflows.
• Guide development teams, share best practices, mentor engineers, and lead technical initiatives.
• Implement data quality checks, security measures (RBAC, Unity Catalog), and ensure data lineage.
• Work with cloud services (Azure/AWS), implement CI/CD pipelines (Git, ADO), and manage Databricks clusters.
• Partner with data scientists, analysts, and business stakeholders to translate requirements into technical solutions.
• Expertise in Deep Spark (PySpark/Scala), Delta Lake, Unity Catalog, SQL, Databricks Notebooks/Workflows, Cloud Platforms (Azure/AWS), CI/CD tools, and data modeling.
• Demonstrated experience in leading projects, mentoring teams, architectural design, managing stakeholders, and resolving complex problems.
• Over 5 years of experience in data engineering and software engineering, with substantial exposure to Databricks.

Perks & Benefits

• Competitive compensation package
• Inclusive employee benefits
• Flexibility programs
• Continuous development and encouragement

Line of Service

Advisory

Industry/Sector

Not Applicable

Specialism

Technology Strategy

Management Level

Senior Manager

Job Description & Summary

At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals.

In data analysis at PwC, you will focus on utilising advanced analytical techniques to extract insights from large datasets and drive data-driven decision-making. You will leverage skills in data manipulation, visualisation, and statistical modelling to support clients in solving complex business problems.

PwC Ireland- Bring Your Talent, Grow Your Skills, Unlock Your Potential

Do you want to be a part of a team of technology and digital leaders, experienced in scoping, and delivering large transformation programmes in complex matrix environments?

Our Data & AI Consulting team is dedicated to leveraging technology in order to solve complex business problems for our clients. We work with Ireland’s largest organisations across a wide range of industries including financial services, government, healthcare, technology, media and telecoms.

Do you want to be a part of a community of problem solvers combining human ingenuity, experience and technology innovation to deliver sustained outcomes and build trust?

At PwC, we believe real transformation is Human Led and Tech Powered. Key to delivering this capability and driving our continued growth is our ability to guide clients in their consideration and implementation of leading Data, Analytics & AI Technologies. As we embark on further building on our deep technical and commercial teams, we are hiring for talent with skills in data strategy, governance, and transformation. By joining our Data & AI team you will have the opportunity to help shape the future of our company and help clients deliver bigger, more impactful transformation that lasts.

Purpose-led work you’ll be part of

As the Lead Developer on our Data & AI team, you will spearhead the development and delivery of analytics projects that significantly enhance healthcare outcomes. Your role will be pivotal in developing, leading, reviewing, and managing code with a focus on Databricks technology for complex healthcare systems. Ensuring the highest standards of code development, delivery, data quality, compliance, and reliability will be a core part of your responsibilities. Collaborating with experienced architects, engineers, and junior developers, you will contribute to building a robust data solution. Your efforts will directly support improved decision-making for healthcare providers, patients, and policymakers. By leveraging your expertise in healthcare data standards and best practices, you will be instrumental in advancing projects that positively impact the sector.

Role & Responsibilities:

• Architecture & Design: Developing scalable data solutions, defining data models, designing data lakes/lakehouses, and creating high-level design documents.
• Development & Optimization: Building and optimizing complex data pipelines (both batch and streaming) using PySpark/Scala, Delta Lake, and SQL, as well as automating workflows.
• Leadership & Mentoring: Guiding development teams, sharing best practices, mentoring engineers, and leading technical initiatives.
• Data Governance & Quality: Implementing data quality checks, security measures (RBAC, Unity Catalog), and ensuring data lineage.
• Cloud & DevOps: Working with cloud services (Azure/AWS), implementing CI/CD pipelines (Git, ADO), and managing Databricks clusters.
• Collaboration: Partnering with data scientists, analysts, and business stakeholders to translate requirements into technical solutions.

Your Profile:

• Technical Skills: Expertise in Deep Spark (PySpark/Scala), Delta Lake, Unity Catalog, SQL, Databricks Notebooks/Workflows, Cloud Platforms (Azure/AWS), and CI/CD tools, along with proficiency in data modeling.
• Leadership Abilities: Demonstrated experience in leading projects, mentoring teams, architectural design, managing stakeholders, and resolving complex problems.
• Professional Experience: Over 5 years of experience in data engineering and software engineering, with substantial exposure to Databricks.

Unlock your potential with PwC Ireland

We believe that challenges are better solved together! We’re inspiring and empowering our people to change the world. Powered by the latest technology, you’ll be a part of amazing teams encouraged to collaborate and innovate in a way that creates powerful solutions and makes a positive impact. This purpose-led work, and our continuous development and encouragement, will help unlock your potential and take your career to the next level.

Enjoy PwC’s perks

We reward your impact, and support your wellbeing, through a competitive compensation package, inclusive employee benefits and flexibility programs that will help you thrive in work and life. Learn more about us at Life@PwC. (https://www.pwc.ie/careers-ie/life-at-pwc.html)

Being appreciated for being you

Our most valuable asset is our people and we grow stronger as we learn from one another. We are an equal opportunity employer and we value diversity. We do not discriminate on the basis of race, religion, colour, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status. We believe this so much that we have signed up for the Business in the Community Elevate Pledge. (https://www.bitc.ie/the-leaders-group-on-sustainability/inclusive-workplace-pledge/)

You can learn more about our culture of belonging and explore our range of inclusive programmes, initiatives, employee resource groups and more at www.pwc.ie.

Avoid the confidence gap; you do not have to match all the listed requirements exactly to apply, we’d love to hear from you!

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please email us for more information.

Education

Degrees/Field of Study required:

Degrees/Field of Study preferred:

Certifications

Required Skills

Accepting Feedback, Accepting Feedback, Active Listening, Algorithm Development, Alteryx (Automation Platform), Analytical Thinking, Analytic Research, Big Data, Business Data Analytics, Coaching and Feedback, Communication, Complex Data Analysis, Conducting Research, Creativity, Customer Analysis, Customer Needs Analysis, Dashboard Creation, Data Analysis, Data Analysis Software, Data Collection, Data-Driven Insights, Data Integration, Data Integrity, Data Mining, Data Modeling {+ 46 more}

Optional Skills

Desired Languages

Travel Requirements

Not Specified

Available for Work Visa Sponsorship?

Yes

Government Clearance Required?

No

Job Posting End Date