Outscal Logooutscal logo

Senior Data Engineer

17 Hours ago • 4 Years + • Data Analyst

Job Summary

Job Description

The NewRich Network seeks a part-time Senior Data Engineer (with a full-time option after 3 months) to build data integrations using AWS and an ELK stack. Responsibilities include designing and building applications for data analysis and transformations on large datasets in a Spark-based AWS environment (EMR, S3, Glue, Redshift, Athena). The role involves evaluating pipeline models, collaborating with various teams (product, data research, engineering), prototyping, testing, and deploying solutions. The ideal candidate possesses 4+ years of data engineering experience, strong Python (PySpark) or Scala skills, and experience with ETL architectures, Apache Airflow, Spark, databases, and AWS.
Must have:
  • 4+ years Data Engineering experience
  • Excellent communication skills
  • Experience with Apache Spark (PySpark/Scala)
  • Strong SQL knowledge
  • AWS experience (EMR, S3, Glue, Redshift, Athena)
  • ETL architecture & standards knowledge
  • Python & Data pipeline building experience
Good to have:
  • Experience with Apache Airflow, MongoDB, MySQL
  • Docker or Kubernetes experience
  • Data validation and analysis protocol creation
  • Experience with 1B+ record Spark jobs

Job Details

Description

The NewRich Network Network is looking for a part-time Data Engineer (Option to go Full-Time in 3 Months) with experience in building data integrations using AWS and an ELK technology stack as part of the team's product portfolio.

We are a company that is 100% focused on delivering digital information and solutions to our customers all around the world. Our dev projects are at the core of our organization – and so is our dev team. We aren’t code monkeys here. We’re creators.

Our dev team is also 100% remote. So if you’re looking for a work-from-home opportunity, and you’re passionate about creating platforms for the new world - we want to hear from you!

Requirements

  • Design and build applications that perform data analysis, transformations, aggregations, and other augmentations on large sets of data in a spark-based AWS environment (EMR, S3, Glue, Redshift, Athena)
  • Evaluate various pipeline models, tools, and environments and implement these to push data from our sources through your transformations and finally to our customers
  • Work with product management and data research teams to prototype and test new ideas then take those to production
  • Work in a fast-paced, innovate-and-test environment.

What You'll Do:

  • Collaborate with Data architects, Enterprise architects, Solution consultants and Product engineering teams to gather customer data integration requirements, conceptualize solutions & build required technology stack
  • Collaborate with enterprise customer's engineering team to identify data sources, profile and quantify quality of data sources, develop tools to prepare data and build data pipelines for integrating customer data sources and third party data sources.
  • Develop new features and improve existing data integrations with customer data ecosystem
  • Encourage the team to think out-of-the-box and overcome engineering obstacles while incorporating new innovative design principles.
  • Collaborate with a Project Manager to bill and forecast time for product owner solutions
  • Building data pipelines
  • Reconciling missed data
  • Acquire datasets that align with business needs
  • Develop algorithms to transform data into useful, actionable information
  • Build, test, and maintain database pipeline architectures
  • Collaborate with management to understand company objectives
  • Create new data validation methods and data analysis protocols
  • Ensure compliance with data governance and security policies

What You Need To Succeed:

  • 4 + years experience in Data Engineering
  • Excellent communication and interpersonal skills are a MUST.
  • Bachelor’s degree in Computer Science, Engineering or a related discipline - Preferred but not mandatory
  • 3+ years of experience working on Apache Spark applications using Python (PySpark) or Scala
  • Experience creating spark jobs that work on at least 1 billion records
  • Strong knowledge of ETL architecture and standards
  • Software development experience working with Apache Airflow, Spark, MongoDB, MySQL
  • Strong SQL knowledge
  • Strong command of Python
  • Experience creating data pipelines in a production system
  • Proven experience in building/operating/maintaining fault tolerant and scalable data processing integrations using AWS
  • Experience using Docker or Kubernetes is a plus
  • Ability to identify and resolve problems associated with production grade large scale data processing workflows
  • Experience with crafting and maintaining unit tests and continuous integration.
  • Passion for crafting Intelligent data pipelines that teams love to use
  • Strong capacity to handle numerous projects are a must

Annual Salary

Ranges from 700,000,000 to 900,000,000 COP

Benefits

For a full list of benefits please go to newrich.com/careers

Similar Jobs

Unity - Senior Software Engineer - Frontend

Unity

Seoul, South Korea (On-Site)
9 Months ago
Zoox - Calibration, Localization, and Mapping Internship/Co-op

Zoox

Foster City, California, United States (On-Site)
4 Months ago
Monsters Aliens Robots Zombies - Senior Full Stack Developer

Monsters Aliens Robots Zombies

Toronto, Ontario, Canada (On-Site)
6 Months ago
ByteDance - Software Researcher/Engineer - Applied Research Center (Infrastructure+AI)

ByteDance

San Jose, California, United States (On-Site)
4 Months ago
Paytm - Business Intelligence Analyst -Junior Manager- First Games

Paytm

Noida, Uttar Pradesh, India (On-Site)
4 Months ago
SuperPlay - Game Economist

SuperPlay

Tel Aviv District, Israel (On-Site)
3 Weeks ago
Mobiloitte - Databricks Certified Data Engineers

Mobiloitte

United States (Remote)
2 Weeks ago
King - Corporate Strategy Analyst / Associate

King

London, England, United Kingdom (On-Site)
2 Weeks ago
PlayStation Global - Senior Security Analyst-Data Governance

PlayStation Global

Carlsbad, California, United States (On-Site)
2 Months ago
Warner Bros Games - WB Games Sales Planning & Analysis Intern

Warner Bros Games

Burbank, California, United States (Hybrid)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

NVIDIA - Senior Systems Software Engineer

NVIDIA

Oregon, United States (On-Site)
1 Month ago
Nintendo - Software Engineer (NTD)

Nintendo

Redmond, Washington, United States (On-Site)
8 Months ago
Google - Lead CPU RTL Engineer, Silicon

Google

(On-Site)
3 Months ago
ByteDance - Software Development Engineer - Network Observation

ByteDance

Singapore (On-Site)
4 Months ago
Intel Corporation - Full Stack Software Development Engineer

Intel Corporation

San José, San José Province, Costa Rica (Hybrid)
2 Months ago
Inkittt - Senior Frontend Engineer

Inkittt

San Francisco, California, United States (On-Site)
7 Months ago
ByteDance - Machine Learning Engineer Intern (Knowledge Graph) - 2024 Start (PhD)

ByteDance

Seattle, Washington, United States (On-Site)
4 Months ago
ByteDance - Research Scientist Graduate (AML- AI-for-Science) - 2025 Start (PhD)

ByteDance

Seattle, Washington, United States (On-Site)
4 Months ago
Tencent - Applied Research Scientist

Tencent

London, England, United Kingdom (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Colombia

Anthology  Inc  - Finance Planning & Analysis Manager

Anthology Inc

Colombia (Remote)
1 Month ago
Evolution - Server Administrator

Evolution

Medellín, Antioquia, Colombia (On-Site)
3 Months ago
Unity - Junior Developer Support Engineer, Standard Support

Unity

Pereira, Risaralda, Colombia (On-Site)
3 Months ago
Teravision Games - Lead Programmer

Teravision Games

Bogotá, Bogota, Colombia (Hybrid)
1 Month ago
Nagarro - Staff Engineer - Salesforce Consultant

Nagarro

Colombia (Remote)
3 Weeks ago
Nagarro - Associate Staff Engineer - Data Scientist

Nagarro

Colombia (Remote)
1 Week ago
Nagarro - Associate Principal Consultant, Support Sales

Nagarro

Colombia (Remote)
4 Months ago
The Workshop - Technical Infrastructure Project Manager

The Workshop

Tocancipá, Cundinamarca, Colombia (On-Site)
3 Months ago
N-iX - Senior Fullstack .NET + Angular Engineer

N-iX

Medellín, Antioquia, Colombia (Hybrid)
6 Days ago
N-iX - Senior QA Engineer with Automation Skills (.NET)

N-iX

Colombia (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

Joyteractive - Lead BI Architect

Joyteractive

Poland (Remote)
10 Hours ago
PhonePe - Business Intelligence, Associate Manager

PhonePe

Bengaluru, Karnataka, India (On-Site)
3 Months ago
Anthology  Inc  - Sr. Data Analyst

Anthology Inc

Colombia (On-Site)
2 Months ago
Axinous - Sr. Analyst, Marketing Strategy & Analytics

Axinous

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Microsoft - Principal Data Scientist

Microsoft

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Huuuge Games - BUSINESS ANALYST

Huuuge Games

Bydgoszcz, Kuyavian-Pomeranian Voivodeship, Poland (Hybrid)
6 Months ago
PENN Interactive - Manager, Analytics Reporting

PENN Interactive

Philadelphia, Pennsylvania, United States (Hybrid)
2 Months ago
Vista - Senior Quality Engineer

Vista

Bengaluru, Karnataka, India (On-Site)
4 Months ago
ION - Senior Technical Business Analyst, New York

ION

New York, New York, United States (Hybrid)
3 Months ago
Tesla - Data Engineering Internship - New Product Introduction, BOM Management

Tesla

Brandenburg, Germany (On-Site)
3 Weeks ago

Get notifed when new similar jobs are uploaded