Lead Software Engineer - Big Data Infrastructure

1 Month ago • 7 Years + • DevOps • $152,200 PA - $223,600 PA

Job Summary

Job Description

The Lead Software Engineer - Big Data Infrastructure will develop, scale, and improve large-scale data processing engines (Spark, Flink, Presto, etc.) in both on-premises and cloud environments. Responsibilities include investigating new big data technologies, building cloud-based infrastructure for batch and streaming data, optimizing performance and scalability, advocating engineering best practices, collaborating with other teams on system design, and defining and leading the adoption of best practices. The role requires experience with ETL and ad-hoc query engines, CI/CD, and working with large-scale data systems. Experience with latest data formats (Iceberg, Delta, Hudi) is also crucial. This position involves close collaboration with product managers and other developers.
Must have:
  • 7+ years programming experience
  • Big data component expertise (Hadoop, Spark, Flink, Presto)
  • Experience building big data infrastructure
  • ETL & ad-hoc query engine optimization
  • CI/CD, metrics, security & compliance
  • Experience with latest data formats (Iceberg, Delta, Hudi)
Good to have:
  • Catalog & metadata management
  • Hadoop & containerized technologies expertise
  • Cloud infrastructure experience (Terraform, K8s, IAM, etc.)
  • Managing big data clusters (1000+ nodes)

Job Details

Job Summary:

Disney Entertainment & ESPN Product & Technology

On any given day at Disney Entertainment & ESPN Product & Technology (DEEP&T), we’re reimagining ways to create magical viewing experiences for the world’s most beloved stories while also transforming Disney’s media business for the future. Whether that’s evolving our streaming and digital products in new and immersive ways, powering worldwide advertising and distribution to maximize flexibility and efficiency, or delivering Disney’s unmatched entertainment and sports content, every day is a moment to make a difference to partners and to hundreds of millions of people around the world.

A few reasons why we think you’d love working for Disney Entertainment & ESPN Product &Technology

  • Building the future of Disney’s media business: DE&E Technologists are designing and building the infrastructure that will power Disney’s media, advertising, and distribution businesses for years to come.

  • Reach & Scale: The products and platforms this group builds and operates delight millions of consumers every minute of every day – from Disney+ and Hulu, to ABC News and Entertainment, to ESPN and ESPN+, and much more.

  • Innovation: We develop and execute groundbreaking products and techniques that shape industry norms and enhance how audiences experience sports, entertainment & news.

About Our Team

The Big Data Infrastructure team manages big data services such as Hadoop, Spark, Flink, Presto, Hive, etc. Our services are distributed across the data center and Cloud, supporting a large scale of data amount and thousands of physical resources. We focus on the virtualization of big data environments, cost efficiency, resiliency, and performance.  

The right person for this role should have proven experience with working in mission-critical infrastructure and enjoy building and maintaining large-scale data systems with the challenge of varied requirements and large storage capabilities. If you are someone who enjoys building large-scale big data infrastructure, then this is a great role for you

Responsibilities

  • Develop, scale, and improve in-house/cloud and open-source large-scale data processing engines or framework systems (e.g. Spark, Flink, Presto/Trino, etc).

  • Investigate new big data technology, and apply it to the DisneyStreaming production environment.

  • Build next-gen cloud-based big data infrastructure for batch and streaming data applications, and continuously improve performance, scalability and availability

  • Handle architectural and design considerations such as performance, scalability, reusability, and flexibility issues.

  • Advocate engineering best practices, including the use of design patterns, code review, and automated unit/functional testing.

  • Work together with other engineering teams to influence them on big data system design and optimization.

  • Define and lead the adoption of best practices and processes. Collaborate with senior internal team members and external stakeholders to gather requirements and drive implementation

  • Collaborate efficiently with Product Managers and other developers to build datastores as a service.

  • Collaborate with senior internal team members and external stakeholders to gather requirements and drive implementation.

Basic Qualifications

  • At least 7 years of professional programming and design experience

  • Bigdata-related components (e.g. HDFS, HBase, Yarn, Hive, Spark, Flink, Presto, Impala, Terraform, EKS, Spinnaker, IAM, EMR, and etc)

  • Experience in building in-house big data infrastructure.

  • Experience in developing and optimizing ETL and ad-hoc query engines (e.g. Spark, Flink, Hive, Presto/Trino, GreenPlum)

  • Experience in CICD, fine-tuned metrics, security and compliance enhancement on compute engines

  • Experience in latest data format (Iceberg, Delta, Hudi)

Preferred Qualifications

  • Experience in catalog and metadata management would be a plus

  • Experience in developing and optimizing Hadoop-related and containerized technologies would be a plus (e.g. HDFS, HBase, Yarn, Kubernetes, docker, RocksDB)

  • Demonstrated ability with cloud infrastructure technologies, including Terraform, K8S, IAM, ELB, Ranger, KMS, S3, Glue etc

  • Experience in managing a big data cluster with over 1000 nodes.

Required Education

  • Bachelor's degree in computer science, Information Systems, Software, Electrical or Electronics Engineering, or comparable field of study, and/or equivalent work experience

Additional Information

#DISNEYTECH


The hiring range for this position in Santa Monica, California is $152,200 to $204,100 per year, in Seattle, Washington is $159,500 to $213,900 per year, and in San Francisco, California is $166,800 to $223,600 per year. The base pay actually offered will take into account internal equity and also may vary depending on the candidate’s geographic region, job-related knowledge, skills, and experience among other factors. A bonus and/or long-term incentive units may be provided as part of the compensation package, in addition to the full range of medical, financial, and/or other benefits, dependent on the level and position offered.

Similar Jobs

DraftKings - Senior Data Engineer, Platform

DraftKings

Boston, Massachusetts, United States (On-Site)
1 Month ago
NVIDIA - Solutions Architect, HPC Systems Engineer

NVIDIA

Santa Clara, California, United States (Remote)
1 Month ago
Sail Point - Staff Site Reliability Engineer (Staff SRE)

Sail Point

United States (Remote)
2 Weeks ago
Otherside Entertainment - Senior DevOps Engineer

Otherside Entertainment

United States (Remote)
1 Month ago
RoofStack - Head of Software Development

RoofStack

Istanbul, İstanbul, Türkiye (On-Site)
2 Months ago
Nagarro - Associate Principal Engineer, DevOps

Nagarro

India (Remote)
7 Months ago
Nagarro - Senior Cloud Consultant

Nagarro

Germany (Remote)
2 Months ago
Info Stretch - Lead Data Engineer

Info Stretch

Chennai, Tamil Nadu, India (On-Site)
7 Months ago
UXBERT Labs - Senior Solution Architect (IoT/Bluetooth Integration)

UXBERT Labs

Riyadh, Riyadh Province, Saudi Arabia (Hybrid)
5 Months ago
Rackspace Technology - Sr Big Data Engineer - Oozie and Pig (GCP)

Rackspace Technology

United States (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

bytedance - Senior Software Engineer - Traffic Infrastructure

bytedance

Singapore (On-Site)
7 Months ago
Workato - Senior Infrastructure Engineer

Workato

Barcelona, Catalonia, Spain (On-Site)
2 Weeks ago
Egnyte - Machine Learning Engineer - AI

Egnyte

India (Remote)
2 Months ago
Meta - Production Engineering

Meta

Austin, Texas, United States (Hybrid)
6 Months ago
InfoStretch Corporation - Infrastructure Solutions Architect 3 – Open Source Software

InfoStretch Corporation

Lansing, Michigan, United States (Hybrid)
3 Weeks ago
Workato - Senior Java Engineer (CDC and Data Integration)

Workato

Nicosia, Nicosia, Cyprus (On-Site)
2 Weeks ago
Postman - Senior Backend Engineer, Cloud Platform

Postman

Bengaluru, Karnataka, India (Hybrid)
1 Month ago
Stake logic - Java Back-end Developer

Stake logic

(Remote)
4 Months ago
Unity - Senior Business Intelligence Developer, Trust & Safety Operations

Unity

Montreal, Quebec, Canada (On-Site)
1 Week ago
Glocomms - Infrastructure Security Engineering Manager

Glocomms

United States (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in Washington, United States

fluence - Senior Fire Test Engineer - Battery Energy Storage

fluence

Houston, Texas, United States (Hybrid)
1 Month ago
Headout - Account Executive

Headout

New York, United States (Hybrid)
2 Weeks ago
Whatnot - Software Engineer, Trust & Safety

Whatnot

San Francisco, California, United States (On-Site)
2 Weeks ago
zoox - Senior Software Engineer, Developer Experience

zoox

Foster City, California, United States (On-Site)
7 Months ago
TransUnion - Cyber Claims Adjuster

TransUnion

Chicago, Illinois, United States (Remote)
2 Weeks ago
Marsh McLennan - Senior Employee Benefits (Health & Benefits) Sales Consultant

Marsh McLennan

Los Angeles, California, United States (Hybrid)
2 Weeks ago
CyberArk - Senior DevOps Engineer

CyberArk

United States (On-Site)
2 Weeks ago
Google - Customer Solutions Engineer

Google

New York, New York, United States (On-Site)
1 Month ago
world relief - Sacramento Economic Empowerment Internship - Winter/Spring 2025

world relief

Sacramento, California, United States (Hybrid)
6 Months ago
Apple - Technology and Metrology Engineering Program Manager - Sensors

Apple

Cupertino, California, United States (On-Site)
2 Weeks ago

Get notifed when new similar jobs are uploaded

DevOps Jobs

KBG Blockchain Game Studios - DevOps (Blockchain Gaming)

KBG Blockchain Game Studios

Thành Phố Hồ Chí Minh, Vietnam (On-Site)
10 Months ago
Hashone Careers - Cloud Engineer

Hashone Careers

Gurugram, Haryana, India (Remote)
6 Months ago
Ion - Microsoft System Engineer, Italy

Ion

Italy (Hybrid)
7 Months ago
Polygon Labs - Senior DevOps Engineer

Polygon Labs

United States (Remote)
1 Month ago
Luxoft - Lead Integration and Release Engineer

Luxoft

Bucharest, Bucharest, Romania (On-Site)
6 Months ago
Kefir games - Middle/Senior DevOps Engineer

Kefir games

Cyprus (On-Site)
5 Months ago
Tencent - Tencent Cloud - Senior Cloud Architect (R&D & Solution Design)

Tencent

Singapore (On-Site)
6 Months ago
Animoca Brands - Senior DevOps Engineer

Animoca Brands

Hong Kong (On-Site)
8 Months ago

Get notifed when new similar jobs are uploaded

About The Company

From classic animated features and exhilarating theme park attractions to cutting edge sports coverage, and the hottest shows on television, The Walt Disney Company has been making magic since 1923, creating unforgettable stories that connect with audiences around the world. And we’re just getting started!

The key to our success…. The Cast, Crew, Imagineers and Employees who honor Disney’s rich legacy by stretching the bounds of imagination to create the never-before-seen, bringing unparalleled entertainment experiences to people of all ages. Begin a career that delivers unparalleled creative content and experiences to audiences around the world and just imagine the stories you could be part of…

What is #LifeAtDisney like? It’s a series of magical moments with cast members and employees developing and telling our stories in the most innovative ways. Whether it’s a day spent as a Disney VoluntEAR, or celebrating the release of a new interactive experience, retail product or movie, our days are filled with the knowledge that we are creating entertainment experiences the whole family can enjoy. Follow @DisneyCareers on Facebook, Twitter and Instagram for a peek behind-the-curtain, and discover how you could connect to a world of stories with Disney!

New York, New York, United States (On-Site)

New York, New York, United States (On-Site)

Burbank, California, United States (On-Site)

Celebration, Florida, United States (On-Site)

Buenos Aires, Buenos Aires, Argentina (On-Site)

Seattle, Washington, United States (On-Site)

Santa Monica, California, United States (On-Site)

Glendale, California, United States (On-Site)

Anaheim, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by The Walt Disney Company

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug