Senior PySpark Data Engineer

Job Summary

Synechron is seeking an experienced and technically proficient Senior PySpark Data Engineer to join our data engineering team. In this role, you will be responsible for developing, optimizing, and maintaining large-scale data processing solutions using PySpark. Your expertise will support our organization’s efforts to leverage big data for actionable insights, enabling data-driven decision-making and strategic initiatives.

Software Requirements

Required Skills:

• Proficiency in PySpark
• Familiarity with Hadoop ecosystem components (e.g., HDFS, Hive, Spark SQL)
• Experience with Linux/Unix operating systems
• Data processing tools like Apache Kafka or similar streaming platforms

Preferred Skills:

• Experience with cloud-based big data platforms (e.g., AWS EMR, Azure HDInsight)
• Knowledge of Python (beyond PySpark), Java or Scala relevant to big data applications
• Familiarity with data orchestration tools (e.g., Apache Airflow, Luigi)

Overall Responsibilities

• Design, develop, and optimize scalable data processing pipelines using PySpark.
• Collaborate with data engineers, data scientists, and business analysts to understand data requirements and deliver solutions.
• Implement data transformations, aggregations, and extraction processes to support analytics and reporting.
• Manage large datasets in distributed storage systems, ensuring data integrity, security, and performance.
• Troubleshoot and resolve performance issues within big data workflows.
• Document data processes, architectures, and best practices to promote consistency and knowledge sharing.
• Support data migration and integration efforts across varied platforms.

Strategic Objectives:

• Enable efficient and reliable data processing to meet organizational analytics and reporting needs.
• Maintain high standards of data security, compliance, and operational durability.
• Drive continuous improvement in data workflows and infrastructure.

Performance Outcomes & Expectations:

• Efficient processing of large-scale data workloads with minimum downtime.
• Clear, maintainable, and well-documented code.
• Active participation in team reviews, knowledge transfer, and innovation initiatives.

Technical Skills (By Category)

Programming Languages:

• Required: PySpark (essential); Python (needed for scripting and automation)
• Preferred: Java, Scala

Databases/Data Management:

• Required: Experience with distributed data storage (HDFS, S3, or similar) and data warehousing solutions (Hive, Snowflake)
• Preferred: Experience with NoSQL databases (Cassandra, HBase)

Cloud Technologies:

• Required: Familiarity with deploying and managing big data solutions on cloud platforms such as AWS (EMR), Azure, or GCP
• Preferred: Cloud certifications

Frameworks and Libraries:

• Required: Spark SQL, Spark MLlib (basic familiarity)
• Preferred: Integration with streaming platforms (e.g., Kafka), data validation tools

Development Tools and Methodologies:

• Required: Version control systems (e.g., Git), Agile/Scrum methodologies
• Preferred: CI/CD pipelines, containerization (Docker, Kubernetes)

Security Protocols:

• Optional: Basic understanding of data security practices and compliance standards relevant to big data management

Experience Requirements

• Minimum of 7+ years of experience in big data environments with hands-on PySpark development.
• Proven ability to design and implement large-scale data pipelines.
• Experience working with cloud and on-premises big data architectures.
• Preference for candidates with domain-specific experience in finance, banking, or related sectors.
• Candidates with substantial related experience and strong technical skills in big data, even from different domains, are encouraged to apply.

Day-to-Day Activities

• Develop, test, and deploy PySpark data processing jobs to meet project specifications.
• Collaborate in multi-disciplinary teams during sprint planning, stand-ups, and code reviews.
• Optimize existing data pipelines for performance and scalability.
• Monitor data workflows, troubleshoot issues, and implement fixes.
• Engage with stakeholders to gather new data requirements, ensuring solutions are aligned with business needs.
• Contribute to documentation, standards, and best practices for data engineering processes.
• Support the onboarding of new data sources, including integration and validation.

Decision-Making Authority & Responsibilities:

• Identify performance bottlenecks and propose effective solutions.
• Decide on appropriate data processing approaches based on project requirements.
• Escalate issues that impact project timelines or data integrity.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or related field. Equivalent experience considered.
• Relevant certifications are preferred: Cloudera, Databricks, AWS Certified Data Analytics, or similar.
• Commitment to ongoing professional development in data engineering and big data technologies.
• Demonstrated ability to adapt to evolving data tools and frameworks.

Professional Competencies

• Strong analytical and problem-solving skills, with the ability to model complex data workflows.
• Excellent communication skills to articulate technical solutions to non-technical stakeholders.
• Effective teamwork and collaboration in a multidisciplinary environment.
• Adaptability to new technologies and emerging trends in big data.
• Ability to prioritize tasks effectively and manage time in fast-paced projects.
• Innovation mindset, actively seeking ways to improve data infrastructure and processes.

S​YNECHRON’S DIVERSITY & INCLUSION STATEMENT
 

Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.

All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law.

Candidate Application Notice