IN_Manager_Azure/Databricks_Application Technology-MS Engg_Advisory_Kolkata

Job Description & Summary

At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth.

Those in data and automation at PwC will focus on automating data internally using automation tools or software to streamline data-related processes within the organisation. Your work will involve automating data collection, data analysis, data visualisation, or any other tasks that involve handling and processing data.

Why PWC

At PwC, you will be part of a vibrant community of solvers that leads with trust and creates distinctive outcomes for our clients and communities. This purpose-led and values-driven work, powered by technology in an environment that drives innovation, will enable you to make a tangible impact in the real world. We reward your contributions, support your wellbeing, and offer inclusive benefits, flexibility programmes and mentorship that will help you thrive in work and life. Together, we grow, learn, care, collaborate, and create a future of infinite experiences for each other. Learn more about us.

At PwC, we believe in providing equal employment opportunities, without any discrimination on the grounds of gender, ethnic background, age, disability, marital status, sexual orientation, pregnancy, gender identity or expression, religion or other beliefs, perceived differences and status protected by law. We strive to create an environment where each one of our people can bring their true selves and contribute to their personal growth and the firm’s growth. To enable this, we have zero tolerance for any discrimination and harassment based on the above considerations.

We are looking for a Solution Architect with deep expertise in Databricks Data Engineering and Python-based software development to design, engineer, and deploy secure, scalable data platforms on Azure + Databricks. The role demands hands-on experience in Spark-based transformations, CI/CD automation, and Databricks asset bundle deployment, with a strong grasp of production-grade governance practices.

Responsibilities:

• Databricks Data Engineering
• Build and optimize robust ETL/ELT pipelines using PySpark and Delta Lake
• Design bronze-silver-gold architecture, schema evolution, and partitioning strategies
• Manage Databricks Unity Catalog, lineage, and ACLs for data governance
• Implement data quality checks, validation layers, and performance tuning for large-scale datasets
• Python Software Engineering
• Develop reusable, modular, and testable Python libraries for data ingestion, transformation, and utilities
• Enforce code quality through unit testing, linters, and error handling standards
• Package code into custom wheels, support dependency management, and ensure compatibility across environments
• Databricks Deployment & CI/CD
• Automate workspace deployment using Databricks Asset Bundles (DAB) and workspace files
• Configure CI/CD pipelines via Azure DevOps, GitHub Actions, and Databricks Repos
• Implement deployment promotion workflows across dev, test, and prod
• Ensure audit logging, run tracking, and change traceability for governance and compliance
• Manage secrets, key vault integrations, and network security (private link, VNet injection)

Skill sets

• 5+ years on Azure Data Stack: ADF, ADLS Gen2, Synapse, Event Hub
• 3+ years hands-on with Databricks (Spark, Delta, DLT, Unity Catalog)
• Strong command of Python with production-grade software engineering practices
• Deep experience in CI/CD for Databricks (DAB, CLI, Repos, DevOps, GitHub Actions)
• Strong knowledge of IaC (Terraform, Bicep) for deploying data infrastructure
• Hands-on experience with Git-based workflows, release versioning, rollback production environments.
• Integrate with various databases (e.g., Cosmos DB,) and message queues (e.g., Kafka, eventhub) for seamless backend operations.
• Experience in one or more Python frameworks (Django, Flask)
• Develop and maintain unit and integration tests using frameworks like pytest and unit test to ensure code quality and reliability.
• Awareness of observability, monitoring, and security practices.

Certifications/Credentials

• Databricks Certified Data Engineer Associate / Professional
• Microsoft Certified: Azure Solutions Architect (AZ-305)
• Microsoft Certified: Azure Data Engineer (DP-203)
• Terraform Associate (optional)

Mandatory skill sets:

Azure and Databricks

Preferred skill sets:

Azure and Databricks

Years of experience required:

6-10 yrs

Education qualification:

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

Education

Degrees/Field of Study required: Bachelor of Engineering, Bachelor of Technology

Degrees/Field of Study preferred:

Certifications

Required Skills

Databricks Platform, Microsoft Azure

Optional Skills

Accepting Feedback, Accepting Feedback, Active Listening, Agile Methodology, Alteryx (Automation Platform), Analytical Thinking, Automation, Automation Framework Design and Development, Automation Programming, Automation Solutions, Automation System Efficiency, Business Analysis, Business Performance Management, Business Process Automation (BPA), Business Transformation, C++ Programming Language, Coaching and Feedback, Communication, Configuration Management (CM), Continuous Process Improvement, Creativity, Daily Scrum, Data Analytics, Data Architecture, Data-Driven Insights {+ 39 more}

Desired Languages

Travel Requirements

Available for Work Visa Sponsorship?

Government Clearance Required?

Job Posting End Date