Associate Lead - Data Engineering

2 Months ago • All levels

Software Development & Engineering

Job Description

The Associate Lead - Data Engineering will deliver end-to-end data and analytics capabilities, including data ingest, transformation, science, and visualization. Responsibilities include designing and deploying databases and data pipelines, developing scalable workflows, and optimizing query performance. Candidates should have expert experience in SQL, Python, PySpark, and Python ML libraries, along with knowledge of data and analytics technologies, CI/CD, and AWS services. The role requires solid scripting skills, experience with big data and streaming technologies, and the ability to lead teams, communicate with stakeholders, and work with agile methodologies.

Good To Have:

Other programming languages (R, Scala, SAS, Java)
AWS Solutions Architect / Developer / Data Analytics Specialty certifications
Professional certification
Masters Degree

Must Have:

Deliver end-to-end data and analytics capabilities.
Design and deploy databases and data pipelines.
Develop scalable and fault-tolerant workflows.
Document issues, solutions, findings, and recommendations.
Learn and apply tools and technologies proficiently.
Optimize performance for queries and dashboards.
Develop and deliver clear, compelling briefings.
Analyze client data & systems for requirements.
Test and validate data pipelines, transformations, datasets, reports, and dashboards.
Develop and communicate solutions architectures.
Provide end user support to other data engineers and analysts.
Expert experience in SQL, Python, PySpark, Python ML libraries.
Experience in data and analytics technologies including SQL/NoSQL/Graph databases, ETL, and BI.
Knowledge of CI/CD and related tools.
Experience with AWS services.
Exposure to Snowflake and Airflow.
Solid scripting skills (e.g., bash/shell scripts, Python).
Proven work experience in data streaming technologies.
Proven work experience in Big Data technologies.
Proven work experience in Linux command-line operations.
Proven work experience in networking knowledge.
Ability to lead the team, communicate with business, gather and interpret business requirements.
Experience with agile delivery methodologies using Jira or similar tools.
Experience working with remote teams.
Bachelor Degree in Computer Science relevant field.

Add these skills to join the top 1% applicants for this job

leadership

data-analytics

data-structures

game-texts

agile-development

gitlab

networking

linux

aws

nosql

data-visualization

hadoop

tableau

spark

data-science

ci-cd

python

shell

sql

scala

jira

bash

java

Description

Data Engineer Responsibilities :

Deliver end-to-end data and analytics capabilities, including data ingest, data transformation, data science, and data visualization in collaboration with Data and Analytics stakeholder groups
Design and deploy databases and data pipelines to support analytics projects
Develop scalable and fault-tolerant workflows
Clearly document issues, solutions, findings and recommendations to be shared internally & externally
Learn and apply tools and technologies proficiently, including:
Languages: Python, PySpark, ANSI SQL, Python ML libraries
Frameworks/Platform: Spark, Snowflake, Airflow, Hadoop , Kafka
Cloud Computing: AWS
Tools/Products: PyCharm, Jupyter, Tableau, PowerBI
Performance optimization for queries and dashboards
Develop and deliver clear, compelling briefings to internal and external stakeholders on findings, recommendations, and solutions
Analyze client data & systems to determine whether requirements can be met
Test and validate data pipelines, transformations, datasets, reports, and dashboards built by team
Develop and communicate solutions architectures and present solutions to both business and technical stakeholders
Provide end user support to other data engineers and analysts

Candidate Requirements :

Expert experience in the following[Should have/Good to have]:
SQL, Python, PySpark, Python ML libraries. Other programming languages (R, Scala, SAS, Java, etc.) are a plus
Data and analytics technologies including SQL/NoSQL/Graph databases, ETL, and BI
Knowledge of CI/CD and related tools such as Gitlab, AWS CodeCommit etc.
AWS services including EMR, Glue, Athena, Batch, Lambda CloudWatch, DynamoDB, EC2, CloudFormation, IAM and EDS
Exposure to Snowflake and Airflow.
Solid scripting skills (e.g., bash/shell scripts, Python)
Proven work experience in the following:
Data streaming technologies
Big Data technologies including, Hadoop, Spark, Hive, Teradata, etc.
Linux command-line operations
Networking knowledge (OSI network layers, TCP/IP, virtualization)
Candidate should be able to lead the team, communicate with business, gather and interpret business requirements
Experience with agile delivery methodologies using Jira or similar tools
Experience working with remote teams
AWS Solutions Architect / Developer / Data Analytics Specialty certifications, Professional certification is a plus
Bachelor Degree in Computer Science relevant field, Masters Degree is a plus