Data Engineer

Role Overview

We are seeking a highly skilled Data Engineer with 6+ years of experience to design, develop, and optimize our data pipelines and systems. The ideal candidate will have strong proficiency in Python, extensive knowledge of SQL programming, and a good understanding of DevOps practices. Hands-on experience with AWS S3 or Azure Databricks is essential for this role.

Key Responsibilities:

• Design, build, and maintain efficient, scalable, and reliable data pipelines to support various business needs.
• Develop and optimize ETL processes to integrate data from various structured and unstructured data sources into centralized storage.
• Work with AWS S3 for data storage and management, ensuring scalability and security of data systems.
• Implement data processing workflows and transformations using Azure Databricks.
• Write complex SQL queries for data extraction, transformation, and analysis.
• Monitor and troubleshoot data pipelines and systems to ensure smooth and uninterrupted operation.
• Collaborate with DevOps teams to implement CI/CD pipelines for data solutions, ensuring high availability and reliability.
• Optimize system performance and scalability by implementing best practices in data engineering.
• Ensure data quality, integrity, and compliance with organizational standards and policies.
• Work closely with cross-functional teams, including data scientists, analysts, and software engineers, to deliver high-impact data solutions.

Required Skills and Qualifications:

• Experience: 6+ years in data engineering or a similar role.
• Programming: Proficient in Python, with a strong understanding of its data structures and libraries (e.g., pandas, NumPy, etc.).
• SQL: Advanced skills in SQL programming for complex data manipulation and analysis.
• DevOps: good experience with CI/CD pipelines, version control systems (e.g., Git), and infrastructure-as-code tools.
• Cloud Platforms: Hands-on experience with AWS S3 or Azure Databricks.
• Data Pipelines: Expertise in building, testing, and maintaining ETL processes and workflows.
• Data Formats: Familiarity with data serialization formats such as JSON, Parquet, and Avro.
• Problem-Solving: Strong analytical and troubleshooting skills for debugging data pipeline issues.
• Collaboration: Excellent communication and teamwork skills to work effectively in a cross-functional environment.

Exposure to visualization tools ( PowerBI/Tableau) would be an added advantage.

Work location & Mode - Pune and Bangalore (Hybrid-3 days)