Staff Product Engineer / Product Specialist - Spark SME

4 Months ago • All levels • DevOps

Job Summary

Job Description

Job Details

Position Summary:
We are seeking an Apache Spark - Subject Matter Expert (SME) who will be responsible for designing, optimizing, and scaling Spark-based data processing systems. This role involves hands-on experience in Spark architecture and core functionalities, focusing on building resilient, high-performance distributed data systems. You will collaborate with engineering teams to deliver high-throughput Spark applications and solve complex data challenges in real-time processing, big data analytics, and streaming.

If you’re passionate about working in fast-paced, dynamic environments and want to be part of the cutting edge of data solutions, this role is for you.

We’re looking for someone who can:

    • Design and optimize distributed Spark-based applications, ensuring low-latency, high-throughput performance for big data workloads.
    • Troubleshooting: Provide expert-level troubleshooting for any data or performance issues related to Spark jobs and clusters.
    • Data Processing Expertise: Work extensively with large-scale data pipelines using Spark's core components (Spark SQL, DataFrames, RDDs, Datasets, and structured streaming).
    • Performance Tuning: Conduct deep-dive performance analysis, debugging, and optimization of Spark jobs to reduce processing time and resource consumption.
    • Cluster Management: Collaborate with DevOps and infrastructure teams to manage Spark clusters on platforms like Hadoop/YARN, Kubernetes, or cloud platforms (AWS EMR, GCP Dataproc, etc.).
    • Real-time Data: Design and implement real-time data processing solutions using Apache Spark Streaming or Structured Streaming.

What makes you the right fit for this position:

    • Expert in Apache Spark: In-depth knowledge of Spark architecture, execution models, and the components (Spark Core, Spark SQL, Spark Streaming, etc.)
    • Data Engineering Practices: Solid understanding of ETL pipelines, data partitioning, shuffling, and serialization techniques to optimize Spark jobs.
    • Big Data Ecosystem: Knowledge of related big data technologies such as Hadoop, Hive, Kafka, HDFS, and YARN.
    • Performance Tuning and Debugging: Demonstrated ability to tune Spark jobs, optimize query execution, and troubleshoot performance bottlenecks.
    • Experience with Cloud Platforms: Hands-on experience in running Spark clusters on cloud platforms such as AWS, Azure, or GCP.
    • Containerization & Orchestration: Experience with containerized Spark environments using Docker and Kubernetes is a plus.
undefined

Similar Jobs

GoTo Group - Senior Software Engineer - Engineering Platform

GoTo Group

Gurugram, Haryana, India (On-Site)
3 Months ago
Meta - Production Engineer

Meta

Bellevue, Washington, United States (Remote)
3 Months ago
ION - Cloud Engineer

ION

Budapest, Hungary (On-Site)
3 Months ago
Gravitee - Senior Full Stack Engineer

Gravitee

Delhi, India (Remote)
4 Months ago
Nielsen Holdings - Software Engineer - ( Java and GO, AWS, Kubernetes, Terraform, Cassandra, PostgreSQL)

Nielsen Holdings

Bengaluru, Karnataka, India (Hybrid)
4 Months ago
ION - Senior DevSecOps Engineer, Italy

ION

Collecchio, Emilia-Romagna, Italy (On-Site)
4 Months ago
Luxoft - Infrastructure Engineer with AWS

Luxoft

Poland, Ohio, United States (Remote)
3 Months ago
PhonePe - Product Solution Engineer

PhonePe

Bengaluru, Karnataka, India (On-Site)
3 Months ago
Reality Games - SysAdmin/DevOps

Reality Games

Kraków, Lesser Poland Voivodeship, Poland (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

undefined - Senior Front End Consultant

United States (Remote)
4 Months ago
ION - Senior Security Architect

ION

Pisa, Tuscany, Italy (On-Site)
4 Months ago
PwC - Senior Machine Learning Engineer, AI (Manager)

PwC

Dublin, County Dublin, Ireland (On-Site)
4 Months ago
Blue Yonder - Sr Functional Architect- WMS expertise Preferred

Blue Yonder

Bengaluru, Karnataka, India (On-Site)
4 Months ago
Growe - Middle DevOps Engineer

Growe

Warsaw, Masovian Voivodeship, Poland (On-Site)
4 Months ago
GoMotive - Senior Computer Vision Engineer

GoMotive

India (Remote)
4 Months ago
Brightly - UI Architect

Brightly

Noida, Uttar Pradesh, India (Hybrid)
5 Months ago
Hedra - Full-Stack Engineer

Hedra

San Francisco, California, United States (On-Site)
5 Months ago
Wind River Systems - Member of Technical Staff - DevOps Framework Engineer

Wind River Systems

Bengaluru, Karnataka, India (On-Site)
3 Months ago
ASSIST Software - Azure DevOps Engineer

ASSIST Software

(Remote)
3 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Bengaluru, Karnataka, India

Saviynt - Senior Engineer I, Software Engineering

Saviynt

Bengaluru, Karnataka, India (Hybrid)
4 Months ago
Zeta - Lead Site Reliability Engineer

Zeta

Bengaluru, Karnataka, India (On-Site)
4 Months ago
WeKan Enterprise Solutions - Architect

WeKan Enterprise Solutions

Bengaluru, Karnataka, India (Hybrid)
5 Months ago
Paytm - Manager-Compliance

Paytm

Noida, Uttar Pradesh, India (On-Site)
4 Months ago
Google - Engineering Manager, Android GPU Driver Quality

Google

Bengaluru, Karnataka, India (On-Site)
3 Months ago
Advarra - Product Support Specialist II

Advarra

Bengaluru, Karnataka, India (Hybrid)
4 Months ago
Cision - Senior Demand Analyst

Cision

India (Remote)
3 Months ago
AiDash - Staff Machine Learning Engineer

AiDash

Bengaluru, Karnataka, India (Hybrid)
4 Months ago
IT Gurus Software - ETL Test Automation Engineer (ETL Tester)

IT Gurus Software

Pune, Maharashtra, India (On-Site)
4 Months ago
Google - Security Engineer, Cloud Threat and Abuse Detection

Google

Bengaluru, Karnataka, India (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

DevOps Jobs

Google - Technical Solutions Engineer, Apigee

Google

(On-Site)
3 Months ago
Nielsen Holdings - Senior Software Engineer - Bigdata ( Java / Scala / Python , Spark, SQL , AWS)

Nielsen Holdings

Mumbai, Maharashtra, India (Hybrid)
4 Months ago
Zelis - Senior Cloud Engineer

Zelis

Hyderabad, Telangana, India (On-Site)
5 Months ago
Luxoft - L2 Production Support Engineer

Luxoft

Bengaluru, Karnataka, India (On-Site)
3 Months ago
Razer - Lead Site Reliability Engineer

Razer

Shanghai, Shanghai, China (On-Site)
4 Months ago
Google - Software Engineer II, Site Reliability Engineering, Cloud Console

Google

Warsaw, Masovian Voivodeship, Poland (On-Site)
3 Months ago
Luxoft - Senior Java Developer

Luxoft

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)
3 Months ago
Ubisoft - Machine Learning Developer

Ubisoft

Montreal, Quebec, Canada (On-Site)
3 Months ago
Wargaming - DevOps Engineer

Wargaming

Belgrade, Serbia (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

United States (Remote)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

View All Jobs

Get notified when new jobs are added by Acceldata

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug