Big Data Lead Engineer — Hadoop, Spark & GCP (BigQuery/DataProc) with Java & Python Expertise

Job Summary

Synechron is seeking an experienced Big Data Lead Engineer to drive the design, development, and management of large-scale data solutions. This full-time leadership role involves guiding a team of data engineers and architects to deliver scalable, efficient, and innovative big data initiatives aligned with organizational and business objectives. The role aims to enable data-driven decision-making, optimize data processing workflows, and stay ahead of emerging industry trends to enhance the company's technological capabilities.

Software Requirements

Candidates must possess strong proficiency with big data technologies such as Hadoop, Spark, and NoSQL databases, with proven experience deploying and managing these tools in enterprise environments. Familiarity with data warehousing, ETL processes, and data visualization tools is required. Hands-on experience with cloud platforms, specifically Google Cloud Platform (GCP), is essential, including managing cloud-based data processing tools like DataProc and BigQuery.

Preferred skills include experience with data pipeline orchestration tools, automation frameworks, and advanced analytics solutions. Knowledge of scripting languages such as Python, shell scripting, and familiarity with security and data governance practices will be advantageous.

Overall Responsibilities

• Lead a team of big data developers and architects to deliver high-quality data solutions, providing technical guidance, mentorship, and support.
• Develop and execute a comprehensive big data strategy that supports business goals, ensuring alignment with organizational priorities.
• Oversee project delivery, ensuring timely, within-budget execution that meets or exceeds stakeholder expectations.
• Design, implement, and optimize scalable big data architectures leveraging Hadoop, Spark, and NoSQL technologies.
• Collaborate with cross-functional teams including product, technology, and marketing to gather requirements and translate them into technical solutions.
• Stay current with evolving big data trends and emerging technologies, assessing their applicability to improve existing workflows.
• Manage multiple projects, balancing stakeholder needs with team capacity, and prioritize efforts to maximize impact.
• Foster a culture of continuous learning, professional growth, and innovation within the team.

Technical Skills (By Category)

Programming Languages:

Required: Proficiency in Java and Python for developing applications, workflows, and automations.

Preferred: Knowledge of scripting languages such as Shell or Perl to support automation and deployment tasks.

Databases/Data Management:

Extensive experience with Hadoop ecosystem components (HDFS, Hive, Spark) and NoSQL databases like Cassandra or MongoDB. Strong capabilities with cloud data warehouses such as BigQuery and Cloud SQL.

Cloud Technologies:

Deep understanding of Google Cloud Platform (GCP), including DataProc, BigQuery, Cloud Storage, and related cloud services for data management and processing.

Frameworks and Libraries:

Hands-on experience with Apache Spark, Hive, and other big data processing libraries; familiarity with data serialization formats like Parquet or Avro.

Development Tools and Methodologies:

Expertise in version control tools such as Git, Agile development practices, and automated CI/CD pipelines to promote efficient workflow management.

Security Protocols:

Knowledge of data security, access control, and compliance standards relevant to cloud data environments to ensure data protection.

Experience Requirements

• Minimum of 7+ years of experience in big data and related software development roles.
• Real-world leadership experience managing teams of data engineers and architects for at least 5 years.
• Proven track record delivering large-scale big data projects within enterprise or consulting environments.
• Strong background in designing scalable architectures with Hadoop, Spark, and NoSQL databases.
• Industry experience in finance, healthcare, retail, or technology sectors is preferred but not mandatory.
• Alternative candidates may demonstrate extensive hands-on experience in data engineering, cloud data migrations, or enterprise analytics projects.

Day-to-Day Activities

• Lead design and implementation of big data solutions, including architecture development and data pipeline creation.
• Mentor team members through code reviews, technical mentoring, and knowledge-sharing sessions.
• Collaborate with stakeholders to clarify requirements, define technical specifications, and ensure solutions meet business needs.
• Monitor project progress, manage workload, and address technical challenges promptly.
• Ensure data solutions are optimized for performance, reliability, and security.
• Stay updated on industry trends and evaluate new tools or processes that could improve project outcomes.
• Coordinate across multidisciplinary teams to achieve project milestones and deliverables.
• Document technical architecture, processes, and best practices for team capacity building and future reference.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or a related technical discipline.
• Certifications in big data technologies (e.g., Cloudera Certified Professional, Databricks, or GCP certifications) are preferred.
• Proven experience designing and delivering large enterprise data solutions.
• Strong commitment to continuous learning and professional development.
• Excellent problem-solving, analytical, and troubleshooting skills.

Professional Competencies

• Critical thinking to analyze complex data challenges and develop innovative solutions.
• Leadership qualities to inspire, mentor, and develop technical talent.
• Effective communication skills, capable of explaining complicated technical concepts to non-technical stakeholders.
• Strong collaboration and stakeholder management abilities.
• Adaptability to rapidly evolving technological trends and project demands.
• Ability to manage multiple priorities and deliver results within deadlines.
• Forward-thinking mindset focused on sustainable, scalable, and efficient analytics solutions.