Senior Data Engineer

1 Month ago • 4 Years + • Data Analyst

Job Summary

Job Description

The NewRich Network seeks a part-time Senior Data Engineer (with a full-time option after 3 months) to build data integrations using AWS and an ELK stack. Responsibilities include designing and building applications for data analysis and transformation on large datasets in a Spark-based AWS environment (EMR, S3, Glue, Redshift, Athena). The role involves evaluating pipeline models, collaborating with various teams (product management, data research, engineering), prototyping, testing, and deploying solutions. Experience with Apache Spark (Python/Scala), ETL architecture, Apache Airflow, SQL, Python, and AWS is crucial. The ideal candidate will be proficient in building and maintaining fault-tolerant, scalable data processing integrations, crafting unit tests, and ensuring data governance and security compliance.
Must have:
  • 4+ years Data Engineering experience
  • Excellent communication skills
  • 3+ years Apache Spark experience (Python/Scala)
  • Strong SQL & Python knowledge
  • AWS data pipeline experience
  • ETL architecture expertise
Good to have:
  • Docker/Kubernetes experience
  • Experience with 1 billion+ record Spark jobs
  • Data validation and analysis protocol creation

Job Details

Description

The NewRich Network Network is looking for a part-time Data Engineer (Option to go Full-Time in 3 Months) with experience in building data integrations using AWS and an ELK technology stack as part of the team's product portfolio.

We are a company that is 100% focused on delivering digital information and solutions to our customers all around the world. Our dev projects are at the core of our organization – and so is our dev team. We aren’t code monkeys here. We’re creators.

Our dev team is also 100% remote. So if you’re looking for a work-from-home opportunity, and you’re passionate about creating platforms for the new world - we want to hear from you!

Requirements

  • Design and build applications that perform data analysis, transformations, aggregations, and other augmentations on large sets of data in a spark-based AWS environment (EMR, S3, Glue, Redshift, Athena)
  • Evaluate various pipeline models, tools, and environments and implement these to push data from our sources through your transformations and finally to our customers
  • Work with product management and data research teams to prototype and test new ideas then take those to production
  • Work in a fast-paced, innovate-and-test environment.

What You'll Do:

  • Collaborate with Data architects, Enterprise architects, Solution consultants and Product engineering teams to gather customer data integration requirements, conceptualize solutions & build required technology stack
  • Collaborate with enterprise customer's engineering team to identify data sources, profile and quantify quality of data sources, develop tools to prepare data and build data pipelines for integrating customer data sources and third party data sources.
  • Develop new features and improve existing data integrations with customer data ecosystem
  • Encourage the team to think out-of-the-box and overcome engineering obstacles while incorporating new innovative design principles.
  • Collaborate with a Project Manager to bill and forecast time for product owner solutions
  • Building data pipelines
  • Reconciling missed data
  • Acquire datasets that align with business needs
  • Develop algorithms to transform data into useful, actionable information
  • Build, test, and maintain database pipeline architectures
  • Collaborate with management to understand company objectives
  • Create new data validation methods and data analysis protocols
  • Ensure compliance with data governance and security policies

What You Need To Succeed:

  • 4 + years experience in Data Engineering
  • Excellent communication and interpersonal skills are a MUST.
  • Bachelor’s degree in Computer Science, Engineering or a related discipline - Preferred but not mandatory
  • 3+ years of experience working on Apache Spark applications using Python (PySpark) or Scala
  • Experience creating spark jobs that work on at least 1 billion records
  • Strong knowledge of ETL architecture and standards
  • Software development experience working with Apache Airflow, Spark, MongoDB, MySQL
  • Strong SQL knowledge
  • Strong command of Python
  • Experience creating data pipelines in a production system
  • Proven experience in building/operating/maintaining fault tolerant and scalable data processing integrations using AWS
  • Experience using Docker or Kubernetes is a plus
  • Ability to identify and resolve problems associated with production grade large scale data processing workflows
  • Experience with crafting and maintaining unit tests and continuous integration.
  • Passion for crafting Intelligent data pipelines that teams love to use
  • Strong capacity to handle numerous projects are a must

Annual Salary

Monthly Ranges from 9,000,000 to 12,000,000 COP

Benefits

For a full list of benefits please go to newrich.com/careers

Similar Jobs

Google - Software Engineer II, Infrastructure, Google Cloud

Google

Hyderabad, Telangana, India (On-Site)
5 Months ago
NVIDIA - Senior Network Algorithms Architect

NVIDIA

Yokne'am Illit, North District, Israel (On-Site)
3 Months ago
Vahanai - Ai Engineer

Vahanai

Bengaluru, Karnataka, India (On-Site)
5 Months ago
Salesforce - Software Engineer Backend

Salesforce

Denver, Colorado, United States (Remote)
1 Month ago
GoMotive - Embedded Engineer

GoMotive

India (Remote)
2 Months ago
InMobiInMobi - Senior Product Analyst

InMobiInMobi

Bengaluru, Karnataka, India (On-Site)
1 Month ago
RoofStack - Mobile Banking Senior Business Analyst

RoofStack

İstanbul, İstanbul, Türkiye (Hybrid)
3 Months ago
Bungie - Senior Product Security Analyst

Bungie

(Hybrid)
3 Months ago
ByteDance - Credit Loan Collection Manager - Global Payment

ByteDance

Jakarta, Jakarta, Indonesia (On-Site)
5 Months ago
Ubisoft - Senior ML Data Scientist

Ubisoft

Montreal, Quebec, Canada (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

ByteDance - Video Analysis and Quality Algorithm Intern 2023 Summer/Fall (MS)

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
Meta - Research Scientist Intern, Systems ML - SW/HW Co-Design - Inference

Meta

Menlo Park, California, United States (On-Site)
5 Months ago
Google - Software Engineer III, Infrastructure, Google Cloud Compute Infrastructure

Google

Kirkland, Washington, United States (On-Site)
3 Months ago
Blind Squirrel Games - Senior Generalist Engineer

Blind Squirrel Games

Austin, Texas, United States (Hybrid)
3 Months ago
THE GAME - Social Creative Automotive

THE GAME

Munich, Bavaria, Germany (Hybrid)
1 Month ago
Tesla - Senior Machine Learning, AI Engineer

Tesla

Brandenburg, Germany (On-Site)
2 Months ago
Trend Micro - NLP / Prompt Engineer (VicOne_Automotive Security)

Trend Micro

Taipei City, Taiwan (On-Site)
6 Months ago
CD PROJEKT RED - Senior Gameplay Programmer

CD PROJEKT RED

Boston, Massachusetts, United States (Hybrid)
2 Months ago
ByteDance - Backend Software Engineer, Data Engineering

ByteDance

Singapore (On-Site)
5 Months ago
NVIDIA - DFX Software Engineer (RDSS Intern)

NVIDIA

Taipei City, Taiwan (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Colombia

Evolution - Accounting Specialist

Evolution

Medellín, Antioquia, Colombia (On-Site)
2 Months ago
N-iX - Senior AQA Engineer

N-iX

Colombia (Remote)
1 Month ago
Salesforce - Lead Employee Success People Consultant

Salesforce

Colombia (Remote)
1 Month ago
Streamline Media Group  Inc  - Level Designer

Streamline Media Group Inc

Bogotá, Bogota, Colombia (On-Site)
6 Months ago
Evolution - Studio Support Specialist (Build Crew)

Evolution

Medellín, Antioquia, Colombia (On-Site)
10 Months ago
Evolution - Project Coordinator Colombia

Evolution

Medellín, Antioquia, Colombia (On-Site)
2 Months ago
Nagarro - Staff Engineer, Java Fullstack

Nagarro

Colombia (Remote)
6 Months ago
Amber - Recruiter (Project Based)

Amber

Bogotá, Bogota, Colombia (On-Site)
5 Months ago
Nagarro - Staff Engineer, QA Automation

Nagarro

Colombia (Remote)
6 Months ago

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

Twitch - Data Engineer - People Analytics

Twitch

San Francisco, California, United States (Hybrid)
1 Month ago
Coinbase - Senior Data Scientist

Coinbase

Hyderabad, Telangana, India (Remote)
6 Months ago
Nintendo - Senior Consumer Insights Analyst

Nintendo

Redmond, Washington, United States (Hybrid)
1 Month ago
Match Group - Sr. Software Engineer, Data

Match Group

Los Angeles, California, United States (Hybrid)
6 Months ago
DraftKings - Analyst II, Compliance

DraftKings

Boston, Massachusetts, United States (On-Site)
1 Month ago
Animoca Brands - Digital Assets Research Analyst

Animoca Brands

Hong Kong (On-Site)
3 Months ago
The Walt Disney Company - Data, Marketing Technology & Web Intern

The Walt Disney Company

Singapore, Singapore (On-Site)
2 Months ago
Behaviour Interactive - Senior Data Developer - Dead by Daylight

Behaviour Interactive

Montreal, Quebec, Canada (Hybrid)
1 Month ago
ION - Internship - Data Science

ION

London, England, United Kingdom (On-Site)
6 Months ago
PwC - Data Architect – Technology Consulting

PwC

Prague, Prague, Czechia (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded