Lead Big Data Engineer (Java, Spark, Cloud, and Data Pipeline)

Job Summary

Synechron is seeking an experienced Lead Big Data Engineer with a strong expertise in Java and Big Data frameworks to define, develop, and optimize scalable data solutions. This role is pivotal in designing high-performance data pipelines, ensuring efficient processing and storage of large datasets, and enabling data-driven decision-making across the organization. The Lead Engineer will provide technical leadership, collaborate across teams, and implement best practices to support the company’s data ecosystem in a modern, cloud-enabled environment.

Purpose:
To architect and deliver reliable, scalable, and secure big data solutions aligned with organizational data strategies.

Value:
This role ensures the organization’s data infrastructure can support advanced analytics, machine learning, and real-time data processing, thereby facilitating business growth and operational excellence.

Software Requirements

Required:

• Proficiency with Java programming (Java 8 or above).
• Strong experience working with Apache Spark (batch and streaming), with proven performance tuning skills.
• Hands-on experience with big data tools such as Hadoop, HBase, Couchbase, and Phoenix.
• Experience deploying and managing data pipelines with CI/CD tools like Jenkins, Bitbucket, GIT, Docker, and OpenShift.
• Knowledge of data modeling, ETL workflows, and data architecture design.
• Familiarity with Linux/Unix environments and scripting.

Preferred:

• Experience with cloud-based big data solutions (e.g., AWS EMR, Azure HDInsight).
• Knowledge of additional data tools like Kafka, Spark SQL, or Delta Lake.

Overall Responsibilities

• Lead the development and optimization of large-scale data pipelines and archive solutions using Java and Spark.
• Design, implement, and maintain robust data infrastructure supporting high-volume data ingestion, transformation, and storage.
• Collaborate with data scientists, analysts, and cross-functional teams to gather requirements, provide technical solutions, and ensure data quality.
• Develop and enforce standards for code quality, performance, security, and scalability.
• Oversee deployment automation and system monitoring to ensure high availability and performance.
• Conduct code reviews, provide technical mentorship, and guide best practices across the data engineering team.
• Keep abreast of emerging big data and cloud technologies, recommending their integration to enhance data platform capabilities.

Strategic Objectives:

• Deliver high-performing, scalable data platforms that underpin analytics and business intelligence initiatives.
• Enhance operational efficiency and data reliability through automation and best practices.
• Promote innovative use of emerging data technologies to unlock new insights.

Performance Outcomes:

• Stable, high-performance data processing pipelines delivering data promptly and accurately.
• Reduced system downtime and optimized resource utilization.
• Effective technical guidance leading to a cohesive, skilled data engineering team.

Technical Skills (By Category)

Programming Languages:

• Essential: Java (Java 8 or higher) for data pipeline development and system integration.
• Preferred: Knowledge of Python or Scala for additional scripting flexibility.

Databases & Data Management:

• Extensive experience working with big data frameworks such as Hadoop, HBase, Couchbase, and Phoenix.
• Understanding of data warehousing, data lakes, and schema design.

Cloud Technologies:

• Hands-on experience deploying big data solutions on cloud platforms such as AWS, Azure, or GCP.
• Knowledge of cloud-native data services and security practices.

Frameworks & Libraries:

• Expertise with Apache Spark (batch and streaming), Spark SQL, and related libraries.
• Familiarity with data serialization formats like Parquet, Avro, or ORC.

Development Tools & Methodologies:

• Proficiency in version control systems like GIT, Jenkins, and containerization tools such as Docker and OpenShift.
• Agile methodologies for iterative delivery and continuous integration.

Security & Data Governance Protocols:

• Knowledge of data security best practices, access controls, and privacy regulations relevant to big data.

Experience Requirements

• Minimum of 5 years as a Data Engineer or related role, with 3+ years in large-scale data pipeline development.
• Demonstrated experience designing, building, and maintaining enterprise-grade big data platforms.
• Proven track record of implementing big data solutions using Java and Spark in a production environment.
• Industry experience in finance, healthcare, retail, or telecom sectors preferred.
• Experience leading technical teams and mentoring junior engineers.

Day-to-Day Activities

• Design, develop, and optimize data pipelines and storage solutions utilizing Java and Spark.
• Collaborate with data analysts, data scientists, and business users to understand requirements and deliver solutions.
• Execute continuous integration and deployment of data solutions to cloud environments.
• Monitor system performance, troubleshoot issues, and implement improvements proactively.
• Conduct code reviews and provide guidance to ensure adherence to best practices.
• Create and update technical documentation, data models, and operational procedures.
• Stay current with industry trends and incorporate new tools or techniques to improve the data platform.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or equivalent.
• Professional certifications related to Big Data (e.g., Cloudera, Hortonworks, AWS Big Data Specialty) or Java development are advantageous.
• Proven ability to lead large scale data projects and drive technical excellence.

Professional Competencies

• Strong analytical and troubleshooting skills for complex data systems.
• Excellent communication skills for stakeholder engagement and documentation.
• Leadership ability with experience mentoring and guiding teams.
• Adaptability to rapidly evolving data technologies and organizational priorities.
• Innovative mindset to leverage emerging platforms and tools for business value.
• Effective time management to prioritize tasks and meet deadlines.

S​YNECHRON’S DIVERSITY & INCLUSION STATEMENT
 

Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.

All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law.

Candidate Application Notice