Technical Lead Data Engineer - Python
Job Summary
Job Description
- 10+ years data engineering experience (2+ years in a lead role)
- Python, Databricks, Azure Data Factory expertise
- Cloud platform experience (AWS, Azure, GCP)
- Data modeling, ETL pipeline development
- Big data technologies (Spark, Hadoop, Kafka)
- SQL and NoSQL database experience
- CI/CD and automation experience
- DevOps and DataOps understanding
- Delta Lake, Lakehouse architecture
- Machine learning platform experience
- Terraform, Kubernetes
- Data governance and compliance (GDPR, CCPA)
- Agile methodologies and Jira
15 skills required
Job Details
Project description
As the Data Engineer Tech Lead, you will be responsible for leading a team of data engineers and overseeing the architecture, development, and optimization of data platforms using Python, Databricks, and cloud-based technologies. You will collaborate with cross-functional teams to ensure data accuracy, scalability, and performance while developing innovative solutions for data processing, analytics, and reporting.
Responsibilities
Team Leadership: Lead and mentor a team of data engineers, providing guidance on best practices in data engineering, code reviews, and design patterns.
Data Pipeline Development: Design, develop, and maintain scalable and efficient data pipelines using Python and Databricks on cloud platforms like AWS, Azure, or GCP.
ETL Processes: Architect and build robust ETL (Extract, Transform, Load) processes to gather, clean, and process large datasets from various data sources.
Data Platform Management: Oversee the management of the data platform, ensuring data integrity, performance optimization, and scalability.
Collaboration: Work closely with data scientists, analysts, and business teams to gather data requirements and translate them into efficient data solutions.
Performance Optimization: Optimize data workflows and Databricks clusters for performance, ensuring minimal latency and maximum efficiency in data processing.
Cloud Integration: Manage cloud-based data infrastructure, implementing best practices for security, scaling, and cost management in cloud environments.
Data Quality & Governance: Ensure data accuracy, consistency, and quality across all pipelines by implementing data validation checks and governance policies.
Automation & CI/CD: Automate data workflows, integrate CI/CD pipelines, and ensure reliable data processing through scheduling, monitoring, and alerting mechanisms.
Documentation: Create and maintain comprehensive documentation of data workflows, pipelines, architecture, and best practices.
Skills
Must have
10+ years of experience in data engineering, with at least 2+ years in a lead role.
Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience).
Strong expertise in Python and experience with Databricks or similar big data platforms plus Azure Data Factory is mandatory.
Solid experience in cloud-based platforms such as AWS, Azure, or Google Cloud, especially with managed data services like Azure Data Lake, AWS S3, Databricks, etc.
Strong understanding of data modeling principles, including data warehousing and relational databases.
Proficiency in building ETL pipelines for batch and real-time data processing.
Hands-on experience with big data technologies (Spark, Hadoop, Kafka, etc.).
Knowledge of working with distributed systems and processing large datasets efficiently.
Familiarity with SQL and non-SQL databases (e.g., PostgreSQL, Cassandra, MongoDB).
Experience with CI/CD pipelines and automation tools for data engineering.
Strong understanding of DevOps and DataOps principles.
Excellent communication, leadership, and problem-solving skills.
Nice to have
Experience with Delta Lake, Lakehouse architecture, or similar data architectures.
Experience with machine learning platforms and integrating data pipelines with ML workflows.
Knowledge of Terraform, Kubernetes, or other infrastructure-as-code tools for cloud infrastructure automation.
Experience in implementing data governance frameworks and compliance with GDPR or CCPA.
Familiarity with Agile methodologies and project management tools such as Jira.
Other
Languages
English: B2 Upper Intermediate
Seniority
Senior
Similar Jobs
HiLabs
Warner Bros Games
PwC
Get notifed when new similar jobs are uploaded
Similar Skill Jobs
Meta
Fabric
Fabric
Beyond Creative
Get notifed when new similar jobs are uploaded
Jobs in Chennai, Tamil Nadu, India
PwC
PwC
Scientific Games
Get notifed when new similar jobs are uploaded
Data Analyst Jobs
Netflix
Universal Music
IGT
Get notifed when new similar jobs are uploaded
About The Company
Luxoft, a DXC Technology Company (NYSE: DXC), is a digital strategy and software engineering firm providing bespoke technology solutions that drive business change for customers the world over. Acquired by U.S. company DXC Technology in 2019, Luxoft is a global operation in 44 cities and 21 countries with an international, agile workforce of nearly 18,000 people. It combines a unique blend of engineering excellence and deep industry expertise, helping over 425 global clients innovate in the areas of automotive, financial services, travel and hospitality, healthcare, life sciences, media and telecommunications.
DXC Technology is a leading Fortune 500 IT services company which helps global companies run their mission critical systems. Together, DXC and Luxoft offer a differentiated customer-value proposition for digital transformation by combining Luxoft’s front-end digital capabilities with DXC’s expertise in IT modernization and integration. Follow our profile for regular updates and insights into technology and business needs.
Get notified when new jobs are added by Luxoft
