Product Support Engineer / Product Specialist - "Hadoop SME"

1 Year ago • All levels • Product Management

Job Summary

Job Description

The Product Support Engineer / Product Specialist - "Hadoop SME" will be responsible for designing, optimizing, and scaling Spark-based data processing systems. This role involves hands-on experience in Spark architecture and core functionalities, focusing on building resilient, high-performance distributed data systems. You will collaborate with engineering teams to deliver high-throughput Spark applications and solve complex data challenges in real-time processing, big data analytics, and streaming. The role requires expertise in troubleshooting, data processing, performance tuning, and cluster management within a big data environment, emphasizing the optimization of Spark jobs and integration with related technologies like Hadoop, Hive, and Kafka.
Must have:
  • Design and optimize distributed Spark-based applications
  • Provide expert-level troubleshooting for Spark jobs and clusters
  • Work extensively with large-scale data pipelines using Spark components
  • Conduct deep-dive performance analysis and debugging of Spark jobs
  • Collaborate with DevOps to manage Spark clusters on various platforms
  • Design and implement real-time data processing solutions
Good to have:
  • Certification in Apache Spark or related big data technologies
  • Experience working with Acceldata's data observability platform
  • Demonstrated experience with scripting languages like Bash and Python
  • Familiarity with concepts related to application, server, and network security
  • Possession of certifications from leading Cloud providers (AWS, Azure, GCP)

Job Details

Position Summary:
We are seeking an Apache Hadoop - Subject Matter Expert (SME) who will be responsible for designing, optimizing, and scaling Spark-based data processing systems. This role involves hands-on experience in Spark architecture and core functionalities, focusing on building resilient, high-performance distributed data systems. You will collaborate with engineering teams to deliver high-throughput Spark applications and solve complex data challenges in real-time processing, big data analytics, and streaming.

If you’re passionate about working in fast-paced, dynamic environments and want to be part of the cutting edge of data solutions, this role is for you.

We’re looking for someone who can:

    • Design and optimize distributed Spark-based applications, ensuring low-latency, high-throughput performance for big data workloads.
    • Troubleshooting: Provide expert-level troubleshooting for any data or performance issues related to Spark jobs and clusters.
    • Data Processing Expertise: Work extensively with large-scale data pipelines using Spark's core components (Spark SQL, DataFrames, RDDs, Datasets, and structured streaming).
    • Performance Tuning: Conduct deep-dive performance analysis, debugging, and optimization of Spark jobs to reduce processing time and resource consumption.
    • Cluster Management: Collaborate with DevOps and infrastructure teams to manage Spark clusters on platforms like Hadoop/YARN, Kubernetes, or cloud platforms (AWS EMR, GCP Dataproc, etc.).
    • Real-time Data: Design and implement real-time data processing solutions using Apache Spark Streaming or Structured Streaming.

What makes you the right fit for this position:

    • Expert in Apache Spark: In-depth knowledge of Spark architecture, execution models, and the components (Spark Core, Spark SQL, Spark Streaming, etc.)
    • Data Engineering Practices: Solid understanding of ETL pipelines, data partitioning, shuffling, and serialization techniques to optimize Spark jobs.
    • Big Data Ecosystem: Knowledge of related big data technologies such as Hadoop, Hive, Kafka, HDFS, and YARN.
    • Performance Tuning and Debugging: Demonstrated ability to tune Spark jobs, optimize query execution, and troubleshoot performance bottlenecks.
    • Experience with Cloud Platforms: Hands-on experience in running Spark clusters on cloud platforms such as AWS, Azure, or GCP.
    • Containerization & Orchestration: Experience with containerized Spark environments using Docker and Kubernetes is a plus.

Good to have:

    • Certification in Apache Spark or related big data technologies.
    • Experience working with Acceldata's data observability platform or similar tools for monitoring Spark jobs.
    • Demonstrated experience with scripting languages like Bash, PowerShell, and Python.
    • Familiarity with concepts related to application, server, and network security management.
    • Possession of certifications from leading Cloud providers (AWS, Azure, GCP), and expertise in Kubernetes would be significant advantages.

Similar Jobs

The E.W. Scripps Company - Senior Analyst, Finance (Hybrid)

The E.W. Scripps Company

Cincinnati, Ohio, United States (Hybrid)
3 Weeks ago
Remote - Benefits Associate, Operations - Southeast Asia

Remote

India (Remote)
3 Weeks ago
Ariens Company - Distribution Center Associate (Tuesday-Friday 7a-5:30) $18/HR

Ariens Company

Shepherdsville, Kentucky, United States (On-Site)
3 Weeks ago
Illumina - Senior Engineer, Product Lifecycle

Illumina

Singapore (On-Site)
1 Month ago
hogarth - Account Executive Intern

hogarth

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)
2 Months ago
Crunchyroll - Senior Producer (Team A)

Crunchyroll

Tokyo, Japan (On-Site)
6 Months ago
Addepar - Product Manager - Android

Addepar

Pune, Maharashtra, India (Hybrid)
3 Months ago
Sabre India - Product Support Analyst I

Sabre India

Kraków, Lesser Poland Voivodeship, Poland (Hybrid)
2 Months ago
undefined - DevOps Production Team Leader

Tel Aviv-Yafo, Tel Aviv District, Israel (Hybrid)
2 Months ago
Cursor - Product Engineer

Cursor

San Francisco, California, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

extreme network - STAFF SW SYSTEMS ENGINEER

extreme network

Bengaluru, Karnataka, India (Hybrid)
3 Months ago
Trend Micro - (Sr.) Cloud Backend Engineer

Trend Micro

Taipei City, Taiwan (On-Site)
10 Months ago
zipdev - Administrative Operations Manager

zipdev

Mexico (Remote)
2 Months ago
Wargaming - Render Engineer

Wargaming

Berlin, Berlin, Germany (On-Site)
2 Months ago
DataVisor - Senior Technical Account Manager (TAM)

DataVisor

Japan (On-Site)
3 Weeks ago
Wind River - Sr. Engineer - Studio Operator - Technical Support Engineer

Wind River

Bengaluru, Karnataka, India (On-Site)
3 Months ago
Rackspace Technology - Senior AWS DevOps Engineer

Rackspace Technology

India (Remote)
2 Months ago
Capgemini - Build Engineer

Capgemini

Bengaluru, Karnataka, India (On-Site)
3 Months ago
Haleon - Regulatory Affairs Oral Health Care Specialist, Export Markets CIS

Haleon

Almaty, Almaty Region, Kazakhstan (Hybrid)
4 Weeks ago
Hawkeye Innovations - Match Day Technician

Hawkeye Innovations

Kilmarnock, Scotland, United Kingdom (On-Site)
4 Weeks ago

Get notifed when new similar jobs are uploaded

Jobs in Bengaluru, Karnataka, India

Western Digital - Enterprise Sales, West

Western Digital

Mumbai, Maharashtra, India (On-Site)
1 Month ago
Capgemini - DevOps Engineer - Splunk & AppDynamics

Capgemini

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Aristocrat - Senior Technical Lead - C++

Aristocrat

Noida, Uttar Pradesh, India (Hybrid)
1 Month ago
Capgemini - SCCM & Intune

Capgemini

Mumbai, Maharashtra, India (On-Site)
3 Months ago
Assystems - Sr Design Engineer

Assystems

Navi Mumbai, Maharashtra, India (On-Site)
9 Months ago
Capgemini - Datacom Developer

Capgemini

Chennai, Tamil Nadu, India (On-Site)
2 Months ago
Philips - Technical Scrum Master

Philips

Bengaluru, Karnataka, India (On-Site)
3 Weeks ago
ShyftLabs - Backend Developer

ShyftLabs

Noida, Uttar Pradesh, India (Hybrid)
3 Weeks ago
Prophecy - Backend Engineer

Prophecy

Bengaluru, Karnataka, India (On-Site)
3 Months ago
xogar games - Unity Developer

xogar games

Bengaluru, Karnataka, India (On-Site)
7 Months ago

Get notifed when new similar jobs are uploaded

Product Management Jobs

Fluxon - Senior Product Manager

Fluxon

Canada (Remote)
3 Months ago
Match Group - Senior Manager, Product Compliance

Match Group

New York, United States (Hybrid)
1 Month ago
Fi - Product Marketing Manager

Fi

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Zynga - Lead Product Manager

Zynga

Barcelona, Catalonia, Spain (Hybrid)
3 Months ago
TransUnion - Specialist III, Product Management

TransUnion

Chennai, Tamil Nadu, India (Hybrid)
3 Weeks ago
Kojima - Producer / Assistant Producer

Kojima

Minato City, Tokyo, Japan (On-Site)
3 Months ago
Toast - Senior Software Engineer (Product-Led Growth - Frontend/Fullstack)

Toast

Dublin, County Dublin, Ireland (Hybrid)
3 Months ago
MiQ - Senior Programmatic Product Manager, Intelligence

MiQ

London, England, United Kingdom (Hybrid)
1 Month ago
Sabre India - Product Support Analyst I

Sabre India

Kraków, Lesser Poland Voivodeship, Poland (Hybrid)
2 Months ago
Thales - Director of Production/Manufacturing and Industrialization

Thales

Ulm, Baden-Württemberg, Germany (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Kitchener, Ontario, Canada (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Bengaluru, Karnataka, India (On-Site)

Campbell, California, United States (Hybrid)

View All Jobs

Get notified when new jobs are added by AccelData

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug