Sr Big Data Engineer (GCP)- Airflow and Oozie

6 Months ago • All levels • DevOps

Job Details

About the Role:
We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing and scaling both stream and batch processing systems, and a solid understanding of public cloud technologies, especially GCP. This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.

What you will be doing
Build a reusable, and reliable code for stream and batch processing systems at scale. This includes working with technologies like Pub/Sub, Kafka, Kinesis, DataFlow, Flink, Hadoop, Pig, Hive, and Spark. Implementing automation/DevOps best practices for CI/CD, IaC, Containerization, etc.

Requirements

    • About the Role:
    • We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing batch processing systems, with extensive experience in Oozie, the Apache Hadoop ecosystem, Airflow, and a solid understanding of public cloud technologies, especially GCP.  This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.

    • What you will be doing
    • Develop scalable and robust code for batch processing systems. This includes working with technologies like Hadoop, Oozie, Pig, Hive, Map Reduce, Spark (Java), Python, Hbase
    • Develop, Manage and optimize data workflows using Oozie and Airflow within the Apache Hadoop ecosystem
    • Leverage GCP for scalable big data processing and storage solutions
    • Implementing automation/DevOps best practices for CI/CD, IaC, etc.

    • Requirements:
    • Experience with GCP managed services and understanding of cloud-based batch processing systems are critical.
    • Proficiency in Oozie, Airflow, Map Reduce, Java
    • Strong programming skills with Java (specifically Spark), Python, Pig, and SQL
    • Expertise in public cloud services, particularly in GCP.
    • Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, Map Reduce
    • Familiarity with BigTable and Redis
    • Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
    • Ability to tackle complex challenges and devise effective solutions. Use critical thinking to approach problems from various angles and propose innovative solutions.
    • Worked effectively in a remote setting, maintaining strong written and verbal communication skills. Collaborate with team members and stakeholders, ensuring clear understanding of technical requirements and project goals.
    • Proven experience in engineering batch processing systems at scale.
    • Hands-on experience in public cloud platforms, particularly GCP. Additional experience with other cloud technologies is advantageous.

undefinedundefined

Similar Jobs

ByteDance - Software Engineer, Architecture and Infrastructure

ByteDance

Seattle, Washington, United States (On-Site)
6 Months ago
Salesforce - PMTS / Principal Software Engineer - Backend - Distributed Systems

Salesforce

Bengaluru, Karnataka, India (On-Site)
7 Months ago
Qventus,  Inc  - Senior Data Engineer

Qventus, Inc

Noida, Uttar Pradesh, India (Hybrid)
5 Months ago
Sinch - Senior Software Engineer (Java)

Sinch

Noida, Uttar Pradesh, India (Hybrid)
6 Months ago
Publicis Groupe - 架构师

Publicis Groupe

Jinan, Shandong, China (On-Site)
6 Months ago
Rackspace Technology - Sr Cloud Architect

Rackspace Technology

India (Remote)
4 Months ago
Microsoft - Software Engineering IC3

Microsoft

Hyderabad, Telangana, India (On-Site)
4 Months ago
Axon - Senior Privacy Engineer

Axon

Scottsdale, Arizona, United States (Hybrid)
5 Months ago
Granicus - Sr. DevOps Engineer

Granicus

Bengaluru, Karnataka, India (Hybrid)
6 Months ago
Kefir Games - Build Engineer

Kefir Games

Cyprus (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

DMarket - Sr. DevOps Engineer/SRE

DMarket

Portugal (Remote)
4 Months ago
PwC - Associate

PwC

Pune, Maharashtra, India (On-Site)
7 Months ago
The Walt Disney Company - Senior Principal Software Engineer

The Walt Disney Company

Santa Monica, California, United States (On-Site)
5 Months ago
PhonePe - Product Solution Engineer

PhonePe

Bengaluru, Karnataka, India (On-Site)
5 Months ago
Centripetal - Cyber Data Scientist

Centripetal

Portsmouth, New Hampshire, United States (On-Site)
8 Months ago
Luxoft - Calypso Dev engineer

Luxoft

New Delhi, Delhi, India (Remote)
4 Months ago
The Walt Disney Company - Sr Software Engineer (Project Hire)

The Walt Disney Company

Seattle, Washington, United States (On-Site)
4 Months ago
Paypal - Principal Machine Learning Engineer - AI

Paypal

San Jose, California, United States (On-Site)
7 Months ago
Luxoft - Full Stack Developer

Luxoft

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)
5 Months ago
Meta - Software Engineer, Intern/Co-op

Meta

Bellevue, Washington, United States (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Jobs in India

Luxoft - Senior Data Engineer/Analyst

Luxoft

Hyderabad, Telangana, India (On-Site)
4 Months ago
PwC - SAP ABAP-Manager

PwC

Kolkata, West Bengal, India (On-Site)
7 Months ago
Rivos - Silicon Logic Formal Verification - Full Time

Rivos

Bengaluru, Karnataka, India (Hybrid)
6 Months ago
PwC - AES SAP Manager (Solution Lead) - Consult

PwC

Bengaluru, Karnataka, India (On-Site)
7 Months ago
Aristocrat Gaming - Chief Software Architect

Aristocrat Gaming

Noida, Uttar Pradesh, India (Hybrid)
6 Months ago
PwC - AES SAP MM-IM Senior Associate  Operate

PwC

Hyderabad, Telangana, India (On-Site)
6 Months ago
Springer Nature - Instructional Design Specialist

Springer Nature

Pune, Maharashtra, India (Hybrid)
6 Months ago
Labcorp - Functional Tester

Labcorp

Bengaluru, Karnataka, India (On-Site)
6 Months ago
SaffronBizz Solutions LLP - 3d Visualizer

SaffronBizz Solutions LLP

Thane, Maharashtra, India (On-Site)
7 Months ago
DViO Digital - Motion Graphic Designer

DViO Digital

Pune, Maharashtra, India (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded

DevOps Jobs

Wargaming - Infrastructure DevOps Engineer

Wargaming

Belgrade, Serbia (Hybrid)
5 Months ago
PwC - Manager- Azure Architect|Bangalore

PwC

Bengaluru, Karnataka, India (On-Site)
7 Months ago
Microsoft - Customer Experience Engineering

Microsoft

(On-Site)
4 Months ago
Google - Cloud Engineer, SAP Analytics

Google

(On-Site)
4 Months ago
IG Group - Senior Systems Engineer

IG Group

Bengaluru, Karnataka, India (Hybrid)
7 Months ago
Luxoft - Senior DevOps (Lambda, Kubernetes)

Luxoft

Poland, Ohio, United States (Remote)
5 Months ago
Info Stretch - Lead Data Engineer

Info Stretch

Pune, Maharashtra, India (On-Site)
6 Months ago
Google - Partner Engineer, Google Cloud

Google

Singapore (On-Site)
4 Months ago
Publicis Groupe - Senior Manager Infrastructure - DevOps GCP/Azure

Publicis Groupe

Bengaluru, Karnataka, India (On-Site)
6 Months ago
Google - Senior Database Blackbelt, Google Cloud

Google

Sunnyvale, California, United States (On-Site)
4 Months ago

Get notifed when new similar jobs are uploaded