Sr Big Data Engineer Airflow and Oozie (GCP)

3 Months ago • All levels • DevOps • $116,100 PA - $198,440 PA

Job Summary

Job Description

This role requires a Senior Big Data Engineer with expertise in developing batch processing systems using Hadoop, Oozie, Airflow, and GCP. Responsibilities include developing scalable code, managing data workflows using Oozie and Airflow, leveraging GCP for data processing and storage, and implementing DevOps best practices (CI/CD, IaC). The ideal candidate will have a strong background in Java (Spark), Python, Pig, SQL, and experience with BigTable and Redis. The position is fully remote and requires excellent communication and problem-solving skills.
Must have:
  • Develop scalable batch processing systems
  • Oozie & Airflow expertise
  • Hadoop ecosystem proficiency
  • GCP experience
  • Java (Spark), Python, Pig, SQL skills
Good to have:
  • BigTable and Redis familiarity
  • DevOps experience (CI/CD, IaC)

Job Details

About the Role:
We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing batch processing systems, with extensive experience in Oozie, the Apache Hadoop ecosystem, Airflow, and a solid understanding of public cloud technologies, especially GCP.  This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.

Work Location: US-Remote

Key Responsibilities:

    • Develop scalable and robust code for batch processing systems. This includes working with technologies like Hadoop, Oozie, Pig, Hive, Map Reduce, Spark (Java), Python, Hbase.
    • Develop, Manage and optimize data workflows using Oozie and Airflow within the Apache Hadoop ecosystem.
    • Leverage GCP for scalable big data processing and storage solutions.
    • Implementing automation/DevOps best practices for CI/CD, IaC, etc.

Qualifications:

    • Bachelors's degree in Computer Science, software engineering or related field of study.
    • Experience with managed cloud services and understanding of cloud-based batch processing systems are critical.
    • Proficiency in Oozie, Airflow, Map Reduce, Java.
    • Strong programming skills with Java (specifically Spark), Python, Pig, and SQL.
    • Expertise in public cloud services, particularly in GCP.
    • Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, Map Reduce.
    • Familiarity with BigTable and Redis.
    • Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
    • Proven experience in engineering batch processing systems at scale.

The following information is required by pay transparency legislation in the following states: CA, CO, HI, NY, and WA. This information applies only to individuals working in these states.
 
·       The anticipated starting pay range for Colorado is: $116,100 - $170,280.
·       The anticipated starting pay range for the states of Hawaii and New York (not including NYC) is: $123,600 - $181,280.
·       The anticipated starting pay range for California, New York City and Washington is: $135,300 - $198,440.

Unless already included in the posted pay range and based on eligibility, the role may include variable compensation in the form of bonus, commissions, or other discretionary payments. These discretionary payments are based on company and/or individual performance and may change at any time. Actual compensation is influenced by a wide array of factors including but not limited to skill set, level of experience, licenses and certifications, and specific work location. Information on benefits  offered is here.
#LI-VM1
#LI-Remote




About Rackspace Technology
We are the multicloud solutions experts. We combine our expertise with the world’s leading technologies — across applications, data and security — to deliver end-to-end solutions. We have a proven record of advising customers based on their business challenges, designing solutions that scale, building and managing those solutions, and optimizing returns into the future. Named a best place to work, year after year according to Fortune, Forbes and Glassdoor, we attract and develop world-class talent. Join us on our mission to embrace technology, empower customers and deliver the future.
 
 
More on Rackspace Technology
Though we’re all different, Rackers thrive through our connection to a central goal: to be a valued member of a winning team on an inspiring mission. We bring our whole selves to work every day. And we embrace the notion that unique perspectives fuel innovation and enable us to best serve our customers and communities around the globe. We welcome you to apply today and want you to know that we are committed to offering equal employment opportunity without regard to age, color, disability, gender reassignment or identity or expression, genetic information, marital or civil partner status, pregnancy or maternity status, military or veteran status, nationality, ethnic or national origin, race, religion or belief, sexual orientation, or any legally protected characteristic. If you have a disability or special need that requires accommodation, please let us know.
 
 

Similar Jobs

InMobiInMobi - Senior Solutions Engineer

InMobiInMobi

London, England, United Kingdom (On-Site)
5 Months ago
Krafton  - IT Service Developer

Krafton

Seoul, South Korea (On-Site)
1 Month ago
Google - Software Engineer II, Google Cloud

Google

(On-Site)
4 Months ago
ByteDance - SRE Software Engineer Intern - Global Payment - 2025 Start

ByteDance

Singapore (On-Site)
2 Months ago
Netflix - Senior Software Engineer, Partner Engineering - APAC

Netflix

Hsinchu, Hsinchu City, Taiwan (On-Site)
6 Months ago
Playgendary - DevOps (Cloud Engineer)

Playgendary

Limassol, Limassol, Cyprus (Remote)
2 Months ago
Zazz - Cloud Engineer (AWS)

Zazz

(Remote)
2 Months ago
PwC - ETIC, Cloud Solution Architect - Manager

PwC

Cairo, Cairo Governorate, Egypt (On-Site)
6 Months ago
PlayStation Global - Senior Build System Engineer

PlayStation Global

United States (Remote)
3 Weeks ago
PENN Interactive - Engineering Manager, ML Platform

PENN Interactive

Philadelphia, Pennsylvania, United States (Hybrid)
2 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Hitachi - Azure Developer

Hitachi

Hyderabad, Telangana, India (Remote)
6 Months ago
Just Play GmbH - Backend Engineer

Just Play GmbH

Berlin, Berlin, Germany (Hybrid)
1 Month ago
Sony Interactive Entertainment - System Development Engineer (PlayStation Platform Game Content Authorship and Delivery System)

Sony Interactive Entertainment

Tokyo, Japan (On-Site)
4 Weeks ago
PwC - Senior Associate _ Automation Tester_ Emerging  Technologies_ Advisory_ Bengaluru

PwC

Bengaluru, Karnataka, India (On-Site)
7 Months ago
Nielsen Holdings - Software Developer - Informatica Power Center / AWS / Oracle

Nielsen Holdings

Bengaluru, Karnataka, India (Hybrid)
6 Months ago
JustPlay - Backend Engineer

JustPlay

Berlin, Berlin, Germany (Hybrid)
1 Month ago
Playrix - Lead SDET

Playrix

Cyprus (Remote)
6 Months ago
ByteDance - Senior Software Engineer (Client Side)

ByteDance

Singapore (On-Site)
3 Months ago
Paytm - Senior QA Engineer - Mobile Apps/SDET

Paytm

Noida, Uttar Pradesh, India (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Jobs in United States

Tencent - Principal Cloud Solution Architect

Tencent

California, United States (On-Site)
1 Month ago
Trackman - Range Sales Manager - Atlantic

Trackman

Stamford, Connecticut, United States (Hybrid)
4 Months ago
The Pokemon Company International - Business Process Analyst

The Pokemon Company International

Redmond, Washington, United States (Hybrid)
1 Month ago
Logitech - Brand Manager, Logitech for Business

Logitech

San Jose, California, United States (Hybrid)
6 Months ago
Light Speed Studios - Art Director

Light Speed Studios

Los Angeles, California, United States (On-Site)
8 Months ago
Meta - Software Engineer, Android

Meta

New York, New York, United States (On-Site)
5 Months ago
ByteDance - Research Scientist Graduate (High-Performance Computing (Algorithm Acceleration)- Vision AI Platform)

ByteDance

San Jose, California, United States (On-Site)
1 Month ago
Onward Search - Payroll Clerk 2

Onward Search

Costa Mesa, California, United States (Hybrid)
2 Months ago
Barracuda Networks  Inc  - XDR Specialist

Barracuda Networks Inc

Dallas, Texas, United States (Remote)
2 Months ago
NVIDIA - Director, Lease Accounting and Portfolio Management

NVIDIA

Santa Clara, California, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

DevOps Jobs

Info Stretch - Programmer Analyst 5

Info Stretch

Lansing, Michigan, United States (Hybrid)
5 Months ago
Playtech - Release Engineer

Playtech

Kyiv, Kyiv City, Ukraine (On-Site)
1 Month ago
Ajmera Infotech - Kubernetes Experts

Ajmera Infotech

Bengaluru, Karnataka, India (On-Site)
10 Months ago
Logifuture - Senior DevOps Engineer

Logifuture

Vojvodina, Serbia (Remote)
1 Month ago
Rackspace Technology - Site Reliability Engineer / Observability Engineer

Rackspace Technology

Giza, Giza Governorate, Egypt (Remote)
4 Months ago
Brillio - Enterprise Architect, Azure - R01535036

Brillio

Bengaluru, Karnataka, India (Hybrid)
6 Months ago
ESL FACEIT Group - EFG - Senior Machine Learning Engineer

ESL FACEIT Group - EFG

(Remote)
4 Weeks ago
White Hat Gaming  - SRE/DevOps Engineer

White Hat Gaming

(Remote)
2 Months ago
Nagarro - Power Platform Developer

Nagarro

Cebu City, Central Visayas, Philippines (On-Site)
6 Months ago
The Walt Disney Company - Senior Platform Engineer - Identity & Access Management

The Walt Disney Company

Orlando, Florida, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded