Data Engineer

Job Summary

Synechron is seeking a motivated and experienced Data Engineer to join our team responsible for designing, developing, and maintaining robust big data solutions. The role involves building scalable data pipelines, managing large datasets, and supporting advanced analytics and data science initiatives. The Data Engineer will play a critical role in enabling data-driven decision-making, ensuring data quality, and optimizing data system performance to meet business objectives.

Software Requirements

Required Skills:

• Proficiency in Big Data frameworks such as Hadoop and Apache Spark (latest stable versions)
• Hands-on experience with NoSQL databases including Cassandra, MongoDB, and HBase
• Knowledge of data ingestion and streaming technologies, particularly Spark Streaming and Apache Flume
• Strong programming skills in Java, Scala, or Python for developing data processing applications
• Experience with DevOps tools such as Git, Jenkins, and Docker for versioning, testing, and deploying data solutions
• Familiarity with cloud platforms like AWS and Azure for data storage, processing, and deployment

Preferred Skills:

• Knowledge of container orchestration tools such as Kubernetes
• Experience with data orchestration tools like Apache Airflow
• Familiarity with data security best practices and encryption techniques in big data environments
• Awareness of machine learning integration with data pipelines

Overall Responsibilities

• Design, develop, and implement scalable big data solutions aligned with business needs and technical standards
• Collaborate cross-functionally to understand data requirements, translating them into optimized technical architectures
• Create and manage data ingestion, transformation, and processing pipelines utilizing Spark Streaming, Flume, and related tools
• Support data scientists and analysts by providing reliable and accessible large datasets for modeling and analytics
• Monitor system health, troubleshoot issues, and optimize performance of big data platforms in production environments
• Ensure data quality, security, and compliance across all data systems
• Engage in continuous improvement by implementing automation, scaling, and performance tuning strategies

Technical Skills (By Category)

• Programming Languages:
• Required: Java, Scala, Python
• Preferred: Additional experience with scripting languages such as Bash or Perl
• Databases/Data Management:
• Required: Experience with NoSQL databases (Cassandra, MongoDB, HBase)
• Preferred: Relational databases and data warehousing solutions such as Snowflake or Redshift
• Cloud Technologies:
• Required: Knowledge of AWS, Azure cloud platforms and services related to data storage and processing
• Preferred: Hands-on experience with cloud-native data tools (e.g., AWS Glue, Azure Data Factory)
• Frameworks and Libraries:
• Hadoop ecosystem components (HDFS, Hive, Pig)
• Spark core, Spark Streaming, and related libraries
• NoSQL-specific SDKs and APIs
• Development Tools and Methodologies:
• Version control with Git (GitLab, GitHub)
• Agile development processes and practices
• CI/CD tools such as Jenkins, Docker, and workflows for automated deployment
• Security & Data Privacy:
• Understanding of data encryption, access controls, and compliance standards applicable to big data environments

Experience Requirements

• 4 to 7 years of professional experience in Big Data roles, with direct hands-on experience in designing and deploying large-scale data solutions
• Proven experience in developing and maintaining data pipelines in cloud or on-premises environments
• Experience working in agile teams and understanding of software development lifecycle (SDLC) practices
• Industry experience across sectors such as finance, healthcare, retail, or technology is advantageous, but solid generalist skills are valued
• Equivalent experience in data integration, pipeline optimization, or related roles can qualify candidates

Day-to-Day Activities

• Lead the development and deployment of data pipelines and systems to support business and analytics needs
• Collaborate with data scientists, analysts, and business stakeholders to gather requirements and translate them into scalable technical solutions
• Design robust workflows for data ingestion from structured and unstructured sources
• Monitor, troubleshoot, and optimize existing data systems to ensure high availability and performance
• Document system architecture, pipelines, and operational procedures for compliance and knowledge sharing
• Automate routine tasks and implement best practices for data system operation and maintenance
• Participate in daily stand-ups, sprint planning, and review meetings within an agile framework

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Science, or a related field
• Relevant certifications (e.g., Cloudera Certified Professional, AWS Certified Data Analytics) are a plus
• Continuous learning in emerging Big Data technologies, data privacy, and industry best practices

Professional Competencies

• Strong analytical and troubleshooting skills to resolve data pipeline issues efficiently
• Effective communication skills for collaborating with technical teams and non-technical stakeholders
• Ability to work independently and as part of a team in a fast-paced environment
• Adaptability to evolving technologies and project requirements
• Attention to detail to ensure data accuracy, security, and compliance
• Demonstrated problem-solving skills with innovative approach to data challenges
• Organizational skills for managing multiple projects and priorities effectively