Lead Data Engineer ( Hadoop) - R01549620
1 Month ago • 5-10 Years
Data Analysis
Job Description
This Senior Data Engineer role involves developing and maintaining Big Data ETL jobs, conducting data analysis and profiling, and collaborating with cross-functional teams to align data engineering solutions with business objectives. The role focuses on automating data workflows, monitoring and optimizing data pipelines for performance and reliability, and ensuring data security and compliance. It offers an opportunity to work with cutting-edge Big Data technologies like Spark, Databricks, and Cloud services.
Good To Have:
- Databricks
- AWS
- Banking data knowledge
- Mastercard data knowledge
Must Have:
- Developing and maintaining Big Data ETL jobs
- Conducting data analysis and profiling
- Automating data workflows
- Monitoring and optimizing data pipelines
- Ensuring data security and compliance
- Proficiency in Python, Java, or Scala
- Experience with Spark and Hadoop frameworks
- Strong SQL skills
- Knowledge of MapReduce, HDFS, Hive
- ETL and Data Warehousing fundamentals
- Data Modelling fundamentals
- Experience with PLSQL, T-SQL, Stored Procedures, Oozie
Primary Skills
- MapReduce, HDFS, Spark - Pyspark, ETL Fundamentals, SQL, SQL (Basic + Advanced), Spark - Scala, Python, Data Warehousing, Hive, Modern Data Platform Fundamentals, Data Modelling Fundamentals, PLSQL, T-SQL, Stored Procedures, Oozie
Job requirements
- Primary Skills (Must Have) : Programming language: Python / Java / Scala, Framework: Spark, Hadoop, Declarative language: SQL Secondary Skills (Good to Have) : Databricks , AWS Scope of Work :
- Developing and maintaining Big Data ETL jobs.
- Conducting data analysis and profiling.
- Collaborating with cross-functional teams (e.g., data scientists, business analysts) to align data engineering solutions with business objectives.
- Automating data workflows to increase system efficiency and reduce manual intervention.
- Monitoring, troubleshooting, and optimizing data pipelines to ensure high performance and reliability.
- Ensuring data security and compliance with Mastercard policies and industry standards.
- Role of Resource : Senior Data Engineer Role USP : Opportunity to work on cutting-edge Big Data technologies like Spark, Databricks, Cloud services with a focus on optimizing and scaling data processing pipelines.
- If Domain knowledge required : Banking / Mastercard data knowledge is an added advantage and preferable.
- Experience Range : 5 to 10 years
Set alerts for more jobs like Lead Data Engineer ( Hadoop) - R01549620
Set alerts for new jobs by Brillio
Set alerts for new Data Analysis jobs in India
Set alerts for new jobs in India
Set alerts for Data Analysis (Remote) jobs
