Senior/Lead Data Engineer

N-iX is looking for a Senior/Lead Data Engineer to join our team!

About the project:

Our customer is the leading school transportation provider in North America, being the owner of more than a half of all yellow school buses in the United States. Every day, the company completes 5 million student journeys, moving more passengers than all U.S. airlines combined and delivers reliable, quality services for 1,100 school districts. 

N-iX has built a successful cooperation with the client delivering a range of complex initiatives. As a result, N-iX has been selected as a strategic long-term partner to drive the digital transformation on an enterprise level, fully remodeling the technology landscape for 55,000 employees and millions of people across North America. 

Responsibilities:

• Design complex ETL processes of various data sources in the data warehouse
• Build new and maintain existing data pipelines using Python to improve efficiency and latency
• Improve data quality through anomaly detection by building and working with internal tools to measure data and automatically detect changes
• Identify, design, and implement internal process improvements, including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes
• Perform data modeling and improve our existing data models for analytics
• Collaborate with SMEs, architects, analysts, and others to build solutions that integrate data from many of our enterprise data sources
• Partner with stakeholders, including data, design, product, and executive teams, and assist them with data-related technical issues

Requirements: 

• Proficiency in Python and SQL
• 3-5 years of commercial experience in building and maintaining a Data Lake
• Good knowledge of AWS cloud services, including the Glue framework
• Experience maintaining Apache Kafka
• Knowledge of job scheduling and orchestration using Airflow
• Well-versed in the optimization of ETL processes
• Steady expertise in data processing tools, including Elasticsearch, Apache Spark, Apache Hive, Apache HBase, Apache Flink, Apache Hudi, Presto.
• Decent knowledge of CI/CD (Docker, Jenkins, Terraform, Git)
• Good understanding of algorithms and data structures
• Excellent communication skills, both written and verbal
• Ukrainian language Advanced or higher.

Nice to have: 

• Experience in the schema and dimensional data design
• Collaboration within a scaled team, using Agile methodology