Lead Engineer

About the Role

We are looking for a highly skilled and self-driven Senior Data Engineer to join our Data Engineering team. The ideal candidate will have deep expertise in building scalable data pipelines and solutions using Databricks, AWS, PySpark, GitHub, and Control-M. This role demands strong experience in ETL project delivery, a solid understanding of data concepts (including data warehouses and lakehouses), and the ability to independently own and deliver complex data engineering tasks.

Key Responsibilities

• Design, develop, and maintain robust data pipelines using PySpark on Databricks.
• Implement ETL workflows and orchestrate them using Control-M.
• Work with AWS services (e.g., S3) to manage data ingestion, transformation, and storage.
• Collaborate with cross-functional teams to understand data requirements and translate them into scalable solutions.
• Apply best practices in version control and CI/CD using GitHub.
• Ensure data quality, integrity, and security across all stages of the pipeline.
• Independently manage delivery timelines, stakeholder communication, and technical decisions.
• Contribute to architectural discussions and mentor junior engineers.
• Leverage AI assistants and automation tools to improve productivity and streamline development workflows.

Required Skills & Experience

• 6+ years of experience in Data Engineering with a strong focus on ETL projects.
• Hands-on experience with Databricks, AWS, PySpark, GitHub, and Control-M.
• Strong understanding of data warehouse and Lakehouse architectures.
• Proven ability to work independently and deliver high-quality solutions.
• Excellent problem-solving and communication skills.
• Experience in Agile/Scrum environments is a plus.

Preferred Qualifications

• Familiarity with AI assistants and their usage in development or delivery workflows.
• Databricks Certification
• Experience with Delta Lake, Apache Spark optimization, and data governance frameworks.
• Exposure to data modeling and performance tuning.