Product Support Engineer / Product Specialist - "Hadoop SME"
Job Summary
Job Description
The Product Support Engineer / Product Specialist - "Hadoop SME" will be responsible for designing, optimizing, and scaling Spark-based data processing systems. This role involves hands-on experience in Spark architecture and core functionalities, focusing on building resilient, high-performance distributed data systems. You will collaborate with engineering teams to deliver high-throughput Spark applications and solve complex data challenges in real-time processing, big data analytics, and streaming. The role requires expertise in troubleshooting, data processing, performance tuning, and cluster management within a big data environment, emphasizing the optimization of Spark jobs and integration with related technologies like Hadoop, Hive, and Kafka.
Must have:
- Design and optimize distributed Spark-based applications
- Provide expert-level troubleshooting for Spark jobs and clusters
- Work extensively with large-scale data pipelines using Spark components
- Conduct deep-dive performance analysis and debugging of Spark jobs
- Collaborate with DevOps to manage Spark clusters on various platforms
- Design and implement real-time data processing solutions
Good to have:
- Certification in Apache Spark or related big data technologies
- Experience working with Acceldata's data observability platform
- Demonstrated experience with scripting languages like Bash and Python
- Familiarity with concepts related to application, server, and network security
- Possession of certifications from leading Cloud providers (AWS, Azure, GCP)
14 skills required
Job Details
Position Summary:
We are seeking an Apache Hadoop - Subject Matter Expert (SME) who will be responsible for designing, optimizing, and scaling Spark-based data processing systems. This role involves hands-on experience in Spark architecture and core functionalities, focusing on building resilient, high-performance distributed data systems. You will collaborate with engineering teams to deliver high-throughput Spark applications and solve complex data challenges in real-time processing, big data analytics, and streaming.
If you’re passionate about working in fast-paced, dynamic environments and want to be part of the cutting edge of data solutions, this role is for you.
We’re looking for someone who can:
- Design and optimize distributed Spark-based applications, ensuring low-latency, high-throughput performance for big data workloads.
- Troubleshooting: Provide expert-level troubleshooting for any data or performance issues related to Spark jobs and clusters.
- Data Processing Expertise: Work extensively with large-scale data pipelines using Spark's core components (Spark SQL, DataFrames, RDDs, Datasets, and structured streaming).
- Performance Tuning: Conduct deep-dive performance analysis, debugging, and optimization of Spark jobs to reduce processing time and resource consumption.
- Cluster Management: Collaborate with DevOps and infrastructure teams to manage Spark clusters on platforms like Hadoop/YARN, Kubernetes, or cloud platforms (AWS EMR, GCP Dataproc, etc.).
- Real-time Data: Design and implement real-time data processing solutions using Apache Spark Streaming or Structured Streaming.
What makes you the right fit for this position:
- Expert in Apache Spark: In-depth knowledge of Spark architecture, execution models, and the components (Spark Core, Spark SQL, Spark Streaming, etc.)
- Data Engineering Practices: Solid understanding of ETL pipelines, data partitioning, shuffling, and serialization techniques to optimize Spark jobs.
- Big Data Ecosystem: Knowledge of related big data technologies such as Hadoop, Hive, Kafka, HDFS, and YARN.
- Performance Tuning and Debugging: Demonstrated ability to tune Spark jobs, optimize query execution, and troubleshoot performance bottlenecks.
- Experience with Cloud Platforms: Hands-on experience in running Spark clusters on cloud platforms such as AWS, Azure, or GCP.
- Containerization & Orchestration: Experience with containerized Spark environments using Docker and Kubernetes is a plus.
Good to have:
- Certification in Apache Spark or related big data technologies.
- Experience working with Acceldata's data observability platform or similar tools for monitoring Spark jobs.
- Demonstrated experience with scripting languages like Bash, PowerShell, and Python.
- Familiarity with concepts related to application, server, and network security management.
- Possession of certifications from leading Cloud providers (AWS, Azure, GCP), and expertise in Kubernetes would be significant advantages.
Similar Jobs
Riot Games
Los Angeles, California, United States (On-Site)
• 3 Months ago
Sonar Source
Austin, Texas, United States (On-Site)
• 2 Months ago
Get notifed when new similar jobs are uploaded
Similar Skill Jobs
Rockstar Games
Carlsbad, California, United States (On-Site)
• 2 Months ago
Get notifed when new similar jobs are uploaded
Jobs in Bengaluru, Karnataka, India
Capgemini
Bengaluru, Karnataka, India (On-Site)
• 1 Month ago
Get notifed when new similar jobs are uploaded
Product Management Jobs
Open Systems Technologies
Raleigh, North Carolina, United States (On-Site)
• 1 Day ago
legion
Bengaluru, Karnataka, India (Hybrid)
• 1 Month ago
Get notifed when new similar jobs are uploaded
