Acceldata

Sales Development Representative - Enterprise

Resident Solutions Architect

Senior SDET - Pulse

Senior SDET - ADOC

<div class="section page-centered" data-qa="job-description"><div><b style="font-size: 10.5pt;">Position Summary:</b></div><div>We are seeking an<b> Apache Spark - Subject Matter Expert (SME)</b> who will be responsible for designing, optimizing, and scaling Spark-based data processing systems. This role involves hands-on experience in Spark architecture and core functionalities, focusing on building resilient, high-performance distributed data systems. You will collaborate with engineering teams to deliver high-throughput Spark applications and solve complex data challenges in real-time processing, big data analytics, and streaming.</div><div><br></div><div>If you’re passionate about working in fast-paced, dynamic environments and want to be part of the cutting edge of data solutions, this role is for you.</div><div><br></div></div><div class="section page-centered"><div><h3>We’re looking for someone who can:</h3><ul class="posting-requirements plain-list"><ul><li>Design and optimize distributed Spark-based applications, ensuring low-latency, high-throughput performance for big data workloads.</li><li><b>Troubleshooting</b>: Provide expert-level troubleshooting for any data or performance issues related to Spark jobs and clusters.</li><li><b>Data Processing Expertise</b>: Work extensively with large-scale data pipelines using Spark's core components (Spark SQL, DataFrames, RDDs, Datasets, and structured streaming).</li><li><b>Performance Tuning</b>: Conduct deep-dive performance analysis, debugging, and optimization of Spark jobs to reduce processing time and resource consumption.</li><li><b>Cluster Management</b>: Collaborate with DevOps and infrastructure teams to manage Spark clusters on platforms like Hadoop/YARN, Kubernetes, or cloud platforms (AWS EMR, GCP Dataproc, etc.).</li><li><b>Real-time Data</b>: Design and implement real-time data processing solutions using Apache Spark Streaming or Structured Streaming.</li></ul></ul></div></div><div class="section page-centered"><div><h3>What makes you the right fit for this position:</h3><ul class="posting-requirements plain-list"><ul><li><b>Expert in Apache Spark</b>: In-depth knowledge of Spark architecture, execution models, and the components (Spark Core, Spark SQL, Spark Streaming, etc.)</li><li><b>Data Engineering Practices</b>: Solid understanding of ETL pipelines, data partitioning, shuffling, and serialization techniques to optimize Spark jobs.</li><li><b>Big Data Ecosystem</b>: Knowledge of related big data technologies such as Hadoop, Hive, Kafka, HDFS, and YARN.</li><li><b>Performance Tuning and Debugging</b>: Demonstrated ability to tune Spark jobs, optimize query execution, and troubleshoot performance bottlenecks.</li><li><b>Experience with Cloud Platforms</b>: Hands-on experience in running Spark clusters on cloud platforms such as AWS, Azure, or GCP.</li><li><b>Containerization &amp; Orchestration</b>: Experience with containerized Spark environments using Docker and Kubernetes is a plus.</li></ul></ul></div></div>undefined