Product Support Engineer / Product Specialist - "Hadoop SME"

1 Year ago • All levels • Product Management

Job Summary

Job Description

The Product Support Engineer / Product Specialist - "Hadoop SME" will be responsible for designing, optimizing, and scaling Spark-based data processing systems. This role involves hands-on experience in Spark architecture and core functionalities, focusing on building resilient, high-performance distributed data systems. You will collaborate with engineering teams to deliver high-throughput Spark applications and solve complex data challenges in real-time processing, big data analytics, and streaming. The role requires expertise in troubleshooting, data processing, performance tuning, and cluster management within a big data environment, emphasizing the optimization of Spark jobs and integration with related technologies like Hadoop, Hive, and Kafka.
Must have:
  • Design and optimize distributed Spark-based applications
  • Provide expert-level troubleshooting for Spark jobs and clusters
  • Work extensively with large-scale data pipelines using Spark components
  • Conduct deep-dive performance analysis and debugging of Spark jobs
  • Collaborate with DevOps to manage Spark clusters on various platforms
  • Design and implement real-time data processing solutions
Good to have:
  • Certification in Apache Spark or related big data technologies
  • Experience working with Acceldata's data observability platform
  • Demonstrated experience with scripting languages like Bash and Python
  • Familiarity with concepts related to application, server, and network security
  • Possession of certifications from leading Cloud providers (AWS, Azure, GCP)

Job Details

Position Summary:
We are seeking an Apache Hadoop - Subject Matter Expert (SME) who will be responsible for designing, optimizing, and scaling Spark-based data processing systems. This role involves hands-on experience in Spark architecture and core functionalities, focusing on building resilient, high-performance distributed data systems. You will collaborate with engineering teams to deliver high-throughput Spark applications and solve complex data challenges in real-time processing, big data analytics, and streaming.

If you’re passionate about working in fast-paced, dynamic environments and want to be part of the cutting edge of data solutions, this role is for you.

We’re looking for someone who can:

    • Design and optimize distributed Spark-based applications, ensuring low-latency, high-throughput performance for big data workloads.
    • Troubleshooting: Provide expert-level troubleshooting for any data or performance issues related to Spark jobs and clusters.
    • Data Processing Expertise: Work extensively with large-scale data pipelines using Spark's core components (Spark SQL, DataFrames, RDDs, Datasets, and structured streaming).
    • Performance Tuning: Conduct deep-dive performance analysis, debugging, and optimization of Spark jobs to reduce processing time and resource consumption.
    • Cluster Management: Collaborate with DevOps and infrastructure teams to manage Spark clusters on platforms like Hadoop/YARN, Kubernetes, or cloud platforms (AWS EMR, GCP Dataproc, etc.).
    • Real-time Data: Design and implement real-time data processing solutions using Apache Spark Streaming or Structured Streaming.

What makes you the right fit for this position:

    • Expert in Apache Spark: In-depth knowledge of Spark architecture, execution models, and the components (Spark Core, Spark SQL, Spark Streaming, etc.)
    • Data Engineering Practices: Solid understanding of ETL pipelines, data partitioning, shuffling, and serialization techniques to optimize Spark jobs.
    • Big Data Ecosystem: Knowledge of related big data technologies such as Hadoop, Hive, Kafka, HDFS, and YARN.
    • Performance Tuning and Debugging: Demonstrated ability to tune Spark jobs, optimize query execution, and troubleshoot performance bottlenecks.
    • Experience with Cloud Platforms: Hands-on experience in running Spark clusters on cloud platforms such as AWS, Azure, or GCP.
    • Containerization & Orchestration: Experience with containerized Spark environments using Docker and Kubernetes is a plus.

Good to have:

    • Certification in Apache Spark or related big data technologies.
    • Experience working with Acceldata's data observability platform or similar tools for monitoring Spark jobs.
    • Demonstrated experience with scripting languages like Bash, PowerShell, and Python.
    • Familiarity with concepts related to application, server, and network security management.
    • Possession of certifications from leading Cloud providers (AWS, Azure, GCP), and expertise in Kubernetes would be significant advantages.

Similar Jobs

SoftSwiss - Ruby on Rails Developer

SoftSwiss

Warsaw, Masovian Voivodeship, Poland (Remote)
2 Weeks ago
WebFX - Jr. Digital Project Specialist

WebFX

Harrisburg, Pennsylvania, United States (On-Site)
8 Months ago
Netflix - HR Business Partner, CEE

Netflix

Warsaw, Masovian Voivodeship, Poland (On-Site)
2 Months ago
Santa Monica Studios - Character Artist (External Development)

Santa Monica Studios

Los Angeles, California, United States (On-Site)
2 Months ago
Evolution  - Team Manager

Evolution

Ezeiza, Buenos Aires Province, Argentina (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Unisys - DBA Vendor Sr Analyst

Unisys

Bogotá, Bogota, Colombia (On-Site)
1 Month ago
Rovio  - Senior Producer, External Projects

Rovio

Barcelona, Catalonia, Spain (Hybrid)
3 Months ago
Qualcomm - Camera Staff Engineer

Qualcomm

Bengaluru, Karnataka, India (On-Site)
1 Month ago
NVIDIA - Senior Optical MSDV Hardware Engineer

NVIDIA

Canada (On-Site)
3 Months ago
Cadence - Principal Software Engineer

Cadence

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Palo Alto Networks - Principal Site Reliability Engineer (Cortex Cloud Security Posture Management)

Palo Alto Networks

Santa Clara, California, United States (On-Site)
2 Weeks ago
smarsh - IT Specialist

smarsh

London, England, United Kingdom (Hybrid)
1 Year ago
Roof Stacks - Senior Frontend Developer

Roof Stacks

İstanbul, Türkiye (On-Site)
1 Month ago
Ariens Company - District Manager-Southwest Florida

Ariens Company

Tampa, Florida, United States (On-Site)
1 Month ago
24bit games - Senior Game Developer

24bit games

(Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in Bengaluru, Karnataka, India

warner bros games - Senior Staff Software Engineer - Fullstack (Content Systems)

warner bros games

Bengaluru, Karnataka, India (Hybrid)
2 Months ago
Haleon - Area Sales Manager

Haleon

Chennai, Tamil Nadu, India (On-Site)
6 Days ago
PwC - Associate|Oracle fusion Technical|Oracle|Advisory|Kolkata’

PwC

Kolkata, West Bengal, India (On-Site)
6 Months ago
Applied materials  - Director of Strategy and Chief of Staff

Applied materials

Bengaluru, Karnataka, India (On-Site)
2 Weeks ago
HCL Tech - Technical Specialist

HCL Tech

Noida, Uttar Pradesh, India (On-Site)
1 Month ago
Google - Technical Lead, Partner Integration

Google

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Capgemini - Order Management

Capgemini

Salem, Tamil Nadu, India (On-Site)
1 Month ago
Alpha Sense - Join AlphaSense India Talent Community

Alpha Sense

Delhi, India (On-Site)
3 Weeks ago
Electronic Arts - Technical Director

Electronic Arts

Hyderabad, Telangana, India (On-Site)
2 Months ago
Interactive Brokers - Director of Facilities- APAC

Interactive Brokers

Mumbai, Maharashtra, India (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Product Management Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

About The Company

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

View All Jobs

Get notified when new jobs are added by AccelData

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug