Senior Data Engineer

1 Month ago • All levels

Job Summary

Job Description

The Senior Data Engineer will be responsible for delivering end-to-end data and analytics capabilities, including data ingestion, transformation, data science, and data visualization. This involves designing and deploying databases and data pipelines to support analytics projects. The engineer will develop scalable and fault-tolerant workflows, clearly document issues and solutions, and apply various tools and technologies proficiently, such as Python, PySpark, SQL, Spark, Snowflake, Airflow, AWS, and others. They will also be expected to optimize performance, develop briefings, analyze client data, and provide support to other data engineers and analysts. The candidate should be able to lead the team, communicate with business, gather and interpret business requirements.
Must have:
  • Expert experience in SQL, Python, and PySpark.
  • Experience with data and analytics technologies like SQL/NoSQL databases.
  • Knowledge of CI/CD tools (Gitlab, AWS CodeCommit).
  • Experience with AWS services (EMR, Glue, Athena, etc.).
  • Solid scripting skills (bash/shell scripts, Python).
  • Experience with data streaming technologies.
  • Experience with Big Data technologies like Hadoop, Spark, Hive, etc.
Good to have:
  • Exposure to Snowflake and Airflow.
  • Other programming languages (R, Scala, SAS, Java, etc.)
  • AWS Solutions Architect / Developer / Data Analytics certifications.

Job Details

Data Engineer Responsibilities :
  • Deliver end-to-end data and analytics capabilities, including data ingest, data transformation, data science, and data visualization in collaboration with Data and Analytics stakeholder groups 
  • Design and deploy databases and data pipelines to support analytics projects 
  • Develop scalable and fault-tolerant workflows 
  • Clearly document issues, solutions, findings and recommendations to be shared internally & externally 
  • Learn and apply tools and technologies proficiently, including: 
    • Languages: Python, PySpark, ANSI SQL, Python ML libraries
    • Frameworks/Platform: Spark, Snowflake, Airflow, Hadoop , Kafka 
    • Cloud Computing: AWS
    • Tools/Products: PyCharm, Jupyter, Tableau, PowerBI 
  • Performance optimization for queries and dashboards
  • Develop and deliver clear, compelling briefings to internal and external stakeholders on findings, recommendations, and solutions 
  • Analyze client data & systems to determine whether requirements can be met 
  • Test and validate data pipelines, transformations, datasets, reports, and dashboards built by team
  • Develop and communicate solutions architectures and present solutions to both business and technical stakeholders
  • Provide end user support to other data engineers and analysts
Candidate Requirements :
  • Expert experience in the following[Should have/Good to have]:
    • SQL, Python, PySpark, Python ML libraries. Other programming languages (R, Scala, SAS, Java, etc.) are a plus
    • Data and analytics technologies including SQL/NoSQL/Graph databases, ETL, and BI
    • Knowledge of CI/CD and related tools such as Gitlab, AWS CodeCommit etc. 
    • AWS services including EMR, Glue, Athena, Batch, Lambda CloudWatch, DynamoDB, EC2, CloudFormation, IAM and EDS
    • Exposure to Snowflake and Airflow.
  • Solid scripting skills (e.g., bash/shell scripts, Python) 
  • Proven work experience in the following:
    • Data streaming technologies
    • Big Data technologies including, Hadoop, Spark, Hive, Teradata, etc.
    • Linux command-line operations
    • Networking knowledge (OSI network layers, TCP/IP, virtualization) 
  • Candidate should be able to lead the team, communicate with business, gather and interpret business requirements
  • Experience with agile delivery methodologies using Jira or similar tools 
  • Experience working with remote teams 
  • AWS Solutions Architect / Developer / Data Analytics Specialty certifications, Professional certification is a plus 
  • Bachelor Degree in Computer Science relevant field, Masters Degree is a plus 

Similar Jobs

Mcdonalds - Learning Designer – HR Curriculum (6 month FTC)

Mcdonalds

London, England, United Kingdom (Hybrid)
2 Days ago
Scopely - Director, People Business Partner

Scopely

London, England, United Kingdom (On-Site)
2 Months ago
caliogo - Vice President, APAC

caliogo

Australia (Remote)
1 Month ago
Glean - Designated Technical Support Engineer

Glean

United States (Remote)
4 Weeks ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Keywords Studios - Senior HR Generalist

Keywords Studios

Austin, Texas, United States (Hybrid)
2 Months ago
ness digital  - QA Engineer with Data expertise

ness digital

Timișoara, Timiș, Romania (Remote)
1 Month ago
Univision - Senior Manager, Talent Management

Univision

Miami, Florida, United States (On-Site)
1 Month ago
The Walt Disney Company - Entertainment Technician (Repair & Maintenance - Props)

The Walt Disney Company

Hong Kong (On-Site)
4 Months ago
Demiurge Studios - Lead Game Designer, Open World RPG

Demiurge Studios

Boston, Massachusetts, United States (Remote)
1 Month ago
Glean - Technical Support Engineer ( Japan)

Glean

Japan (Remote)
5 Months ago
Gearbox - Associate Technical Director

Gearbox

Frisco, Texas, United States (Remote)
6 Months ago
Dave Ramsey - Senior Java Engineer

Dave Ramsey

Franklin, Tennessee, United States (On-Site)
2 Days ago
Glean - Partner Solutions Engineer - EMEA

Glean

United Kingdom (Remote)
4 Weeks ago
Axi - Lead Security Engineer

Axi

Dubai, Dubai, United Arab Emirates (On-Site)
4 Weeks ago

Get notifed when new similar jobs are uploaded

Jobs in Thiruvananthapuram, Kerala, India

Dream Sports - MISE - Sales (Bangalore)

Dream Sports

Bengaluru, Karnataka, India (On-Site)
5 Months ago
sinch  - Lead System Engineer/Team Lead - System & Operations

sinch

Mumbai, Maharashtra, India (On-Site)
1 Week ago
PwC - IN-Associate Internal Audit

PwC

Gurugram, India (On-Site)
1 Week ago
Bosch Group India - S/4 HANA Public cloud FICO consultant

Bosch Group India

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Zamp - General Manager

Zamp

Bengaluru, Karnataka, India (On-Site)
12 Months ago
Drivetrain - SDE (Automation & Quality Focus)

Drivetrain

Kolkata, West Bengal, India (Remote)
10 Months ago
Nagarro - Senior Staff Engineer, Delivery

Nagarro

Bengaluru, Karnataka, India (On-Site)
7 Months ago
Morning Star - Lead Quantitative Analyst

Morning Star

Mumbai, Maharashtra, India (Hybrid)
1 Week ago
Paytm - State Head - AGM / DGM - Udhampur

Paytm

Jammu, Himachal Pradesh, India (On-Site)
6 Months ago
ISS Stoxx - ESG Junior Data Analyst

ISS Stoxx

Mumbai, Maharashtra, India (On-Site)
4 Days ago

Get notifed when new similar jobs are uploaded

Similar Category Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

About The Company

Berwyn, Pennsylvania, United States (On-Site)

Raleigh, North Carolina, United States (Remote)

Thiruvananthapuram, Kerala, India (On-Site)

Raleigh, North Carolina, United States (Remote)

Thiruvananthapuram, Kerala, India (On-Site)

Thiruvananthapuram, Kerala, India (On-Site)

Berwyn, Pennsylvania, United States (Hybrid)

Berwyn, Pennsylvania, United States (On-Site)

Thiruvananthapuram, Kerala, India (On-Site)

Raleigh, North Carolina, United States (Hybrid)

View All Jobs

Get notified when new jobs are added by Yodlee

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug