Jobs

Scholarships

Courses

Companies

Placements

Community

Home >

Jobs >

Technical Lead Data Engineer - Python

Karnataka, India (On-site)

Technical Lead Data Engineer - Python

2 Weeks ago • 10 Years + • Data Analyst • DevOps

About the job

Summary

As the Data Engineer Tech Lead, you will lead a team, oversee data platform architecture, development, and optimization using Python, Databricks, and cloud technologies. Responsibilities include team mentorship, data pipeline development (Python, Databricks), ETL process architecture, data platform management, collaboration with cross-functional teams, performance optimization, cloud integration (AWS, Azure, GCP), data quality assurance, automation (CI/CD), and documentation. The role requires strong expertise in Python, Databricks, cloud platforms, data modeling, ETL processes, big data technologies (Spark, Hadoop, Kafka), SQL/NoSQL databases, and CI/CD. Experience with Delta Lake, Lakehouse architecture, and machine learning platforms are beneficial.

15 skills required for this role

Test your skills to join the top 1% applicants for this job

python

azure

ci-cd

aws

sql

scalability

jira

cassandra

kubernetes

mongodb

spark

terraform

hadoop

design-patterns

postgresql

Must have:

10+ years data engineering experience (2+ years lead role)
Python, Databricks expertise
Cloud platform experience (AWS, Azure, GCP)
Data modeling, ETL pipeline building
Big data technologies (Spark, Hadoop, Kafka)
SQL & NoSQL database knowledge
CI/CD pipeline experience
Excellent communication & leadership

Good to have:

Delta Lake, Lakehouse architecture
Machine learning platform integration
Terraform, Kubernetes experience
Data governance, GDPR/CCPA compliance
Agile methodologies, Jira

Not hearing back from companies?

Unlock the secrets to a successful job application and accelerate your journey to your next opportunity.

Project description

As the Data Engineer Tech Lead, you will be responsible for leading a team of data engineers and overseeing the architecture, development, and optimization of data platforms using Python, Databricks, and cloud-based technologies. You will collaborate with cross-functional teams to ensure data accuracy, scalability, and performance while developing innovative solutions for data processing, analytics, and reporting.

Responsibilities

Team Leadership: Lead and mentor a team of data engineers, providing guidance on best practices in data engineering, code reviews, and design patterns.

Data Pipeline Development: Design, develop, and maintain scalable and efficient data pipelines using Python and Databricks on cloud platforms like AWS, Azure, or GCP.

ETL Processes: Architect and build robust ETL (Extract, Transform, Load) processes to gather, clean, and process large datasets from various data sources.

Data Platform Management: Oversee the management of the data platform, ensuring data integrity, performance optimization, and scalability.

Collaboration: Work closely with data scientists, analysts, and business teams to gather data requirements and translate them into efficient data solutions.

Performance Optimization: Optimize data workflows and Databricks clusters for performance, ensuring minimal latency and maximum efficiency in data processing.

Cloud Integration: Manage cloud-based data infrastructure, implementing best practices for security, scaling, and cost management in cloud environments.

Data Quality & Governance: Ensure data accuracy, consistency, and quality across all pipelines by implementing data validation checks and governance policies.

Automation & CI/CD: Automate data workflows, integrate CI/CD pipelines, and ensure reliable data processing through scheduling, monitoring, and alerting mechanisms.

Documentation: Create and maintain comprehensive documentation of data workflows, pipelines, architecture, and best practices.

Skills

Must have

10+ years of experience in data engineering, with at least 2+ years in a lead role.

Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience).

Strong expertise in Python and experience with Databricks or similar big data platforms plus Azure Data Factory is mandatory.

Solid experience in cloud-based platforms such as AWS, Azure, or Google Cloud, especially with managed data services like Azure Data Lake, AWS S3, Databricks, etc.

Strong understanding of data modeling principles, including data warehousing and relational databases.

Proficiency in building ETL pipelines for batch and real-time data processing.

Hands-on experience with big data technologies (Spark, Hadoop, Kafka, etc.).

Knowledge of working with distributed systems and processing large datasets efficiently.

Familiarity with SQL and non-SQL databases (e.g., PostgreSQL, Cassandra, MongoDB).

Experience with CI/CD pipelines and automation tools for data engineering.

Strong understanding of DevOps and DataOps principles.

Excellent communication, leadership, and problem-solving skills.

Nice to have

Experience with Delta Lake, Lakehouse architecture, or similar data architectures.

Experience with machine learning platforms and integrating data pipelines with ML workflows.

Knowledge of Terraform, Kubernetes, or other infrastructure-as-code tools for cloud infrastructure automation.

Experience in implementing data governance frameworks and compliance with GDPR or CCPA.

Familiarity with Agile methodologies and project management tools such as Jira.

Other

Languages

English: B2 Upper Intermediate

Seniority

Senior

View Full Job Description

About The Company

Luxoft

1206 Active Jobs

Luxoft, a DXC Technology Company (NYSE: DXC), is a digital strategy and software engineering firm providing bespoke technology solutions that drive business change for customers the world over. Acquired by U.S. company DXC Technology in 2019, Luxoft is a global operation in 44 cities and 21 countries with an international, agile workforce of nearly 18,000 people. It combines a unique blend of engineering excellence and deep industry expertise, helping over 425 global clients innovate in the areas of automotive, financial services, travel and hospitality, healthcare, life sciences, media and telecommunications.

DXC Technology is a leading Fortune 500 IT services company which helps global companies run their mission critical systems. Together, DXC and Luxoft offer a differentiated customer-value proposition for digital transformation by combining Luxoft’s front-end digital capabilities with DXC’s expertise in IT modernization and integration. Follow our profile for regular updates and insights into technology and business needs.