Data Engineer

1 Month ago • 4 Years +

Job Summary

Job Description

PayPay is rapidly expanding its product teams, necessitating a robust Data Engineering Platform. The Data as a Service (DaaS) team is responsible for designing, implementing, and operating this platform using technologies like Spark, Hudi, Delta Lake, Scala, and AWS data tools. This role involves designing and maintaining scalable data ingestion pipelines using AWS Glue, Step Functions, Lambda, and Terraform. The role also requires optimizing data pipelines, implementing data processing workflows with Hudi, Delta Lake, Spark, and Scala. The candidate will collaborate with cross-functional teams and implement best practices for observability, data governance, security, and compliance. This role requires the candidate to work in a fast-paced environment and manage multiple tasks simultaneously.
Must have:
  • 4+ years of experience as a Data Engineer.
  • Hands-on experience with Apache Hudi, Delta Lake, Spark, and Scala.
  • Experience in designing and operating a DataLake or Data Warehouse.
  • Expertise in AWS services, including Glue, Step Functions, Lambda, and EMR.
  • Proficiency in Terraform for infrastructure as code (IaC).
Good to have:
  • Knowledge of Data Orchestration tools such as Airflow, Dagster, Prefect.
  • Familiarity with change data capture tools like Canal, Debezium, and Maxwell.
  • Experience with data warehousing tools like AWS Athena, BigQuery, Databricks.
  • Experience in Scala, Python, Java, and SQL.

Job Details

 

 

About PayPay India 

PayPay, a fintech company providing a service enjoyed by over 63 million users (as of April 2024) merely 5 years since its launch in 2018 in Japan. The company is now home to a very diverse team of members from more than 50 countries. We grew to a team of several thousand employees in Japan but are far from over. We are still in the Day 1. Every day, new members join us from all over the world to create new value and deliver it to society.
 
 
 
 

Why India ?

To build our Payment services, we got technical cooperation from Paytm (A large payment service company in India). And based on their customer-first technologies , we created and expanded the smartphone payment service in Japan. Therefore, we have decided to establish a development base in India, because it is a major IT country with many talented engineers, as evidenced by the fact that cutting-edge mobile payments can continue to be generated.

OUR VISION IS UNLIMITED

We dare to believe that we do not need a clear vision to create a future beyond our imagination. PayPay will always stay true to our roots and realise a vision (future) that no one else can imagine by constantly taking risks and challenging ourselves. With this mindset, you will be presented with new and exciting opportunities on a daily basis and have the opportunity to grow and reach new dimensions that you could never have imagined.
 

Job Description

PayPay's growth is driving a rapid expansion of PayPay product teams and the need for a robust Data Engineering Platform to support our growing business needs is more critical than ever. The DaaS team’s responsibility is to design, implement, and operate this platform using cutting edge technologies such as Spark, Hudi, Delta Lake, Scala, and AWS suite of data tools.

We are looking for talented Data Engineers to join our team and help us scale our platform across the organizations.

 

Main Responsibilities

  • Design, develop, and maintain scalable data ingestion pipelines using AWS Glue, Step Functions, Lambda, and Terraform.
  • Optimize and manage large scale data pipelines to ensure high performance, reliability, and efficiency.
  • Implement data processing workflows using Hudi, Delta Lake, Spark, and Scala.
  • Maintain and enhance Lakeformation and Glue Data Catalog for effective data management and discovery.
  • Collaborate with cross-functional teams to ensure seamless data flow and integration across the organization.
  • Implement best practices for observability, data governance, security, and compliance.

Qualifications

  • 4+ years experience as a Data Engineer or in a similar role.
  • Hands-on experience with Apache Hudi, Delta Lake, Spark, and Scala.
  • Experience designing, building, and operating a DataLake or Data Warehouse.
  • Knowledge of Data Orchestration tools such as Airflow, Dagster, Prefect.
  • Strong expertise in AWS services, including Glue, Step Functions, Lambda, and EMR.
  • Familiarity with change data capture tools like Canal, Debezium, and Maxwell.
  • Experience with data warehousing tools like AWS Athena, BigQuery, Databricks.
  • Experience in at least one primary language (e.g. Scala, Python, Java) and SQL (any variant).
  • Experience with data cataloging and metadata management using AWS Glue Data Catalog, Lakeformation, or Unity Catalog.
  • Proficiency in Terraform for infrastructure as code (IaC).
  • Strong problem-solving skills and ability to troubleshoot complex data issues.
  • Excellent communication and collaboration skills.
  • Ability to work in a fast-paced, dynamic environment and manage multiple tasks simultaneously.

 

 

Similar Jobs

Synechron - Lead Java Developer

Synechron

Pune, Maharashtra, India (On-Site)
1 Week ago
Synechron - Senior Data Engineer (AWS, Airflow, DBT & Cloud Data Pipelines)

Synechron

Bengaluru, Karnataka, India (On-Site)
3 Weeks ago
Loyalty Juggernaut - Mobile Engineer (Android)

Loyalty Juggernaut

Hyderabad, Telangana, India (On-Site)
2 Months ago
Google - Cloud Technical Solutions Engineer, Workspace

Google

Tokyo, Japan (On-Site)
1 Month ago
Meta - Research Scientist Intern, Smart Glasses in Wearables AI (PhD)

Meta

Redmond, Washington, United States (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

mighty bear games - Sr. Full Stack Engineer

mighty bear games

Singapore (Remote)
1 Month ago
ISS Stoxx - Software Engineer - Java/UI

ISS Stoxx

Mumbai, Maharashtra, India (On-Site)
3 Weeks ago
koolhaus games - Technical Director Video Games

koolhaus games

Canada (On-Site)
2 Weeks ago
Trend Micro - DevOps Engineer

Trend Micro

Manila, Metro Manila, Philippines (On-Site)
5 Days ago
Salesforce - Senior Data Scientist

Salesforce

Hyderabad, Telangana, India (Hybrid)
4 Days ago
Epic Games - Senior Gameplay Systems Programmer, Fortnite

Epic Games

Canada (On-Site)
1 Month ago
King - Senior Staff Backend Engineer

King

San Francisco, California, United States (On-Site)
3 Days ago
buildstsaff - Full Stack Developer

buildstsaff

Irvine, California, United States (On-Site)
7 Years ago
Crunchyroll - Director of Engineering, Android

Crunchyroll

Culver City, California, United States (On-Site)
3 Months ago
CME Group - Sr QA Analyst

CME Group

Chicago, Illinois, United States (Hybrid)
2 Weeks ago

Get notifed when new similar jobs are uploaded

Jobs in Gurugram, India

Accenture - Application Developer

Accenture

Navi Mumbai, Maharashtra, India (On-Site)
6 Days ago
Qualcomm - Modem hardware modelling -Sr Eng

Qualcomm

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Accenture - Application Developer

Accenture

Hyderabad, Telangana, India (On-Site)
1 Day ago
Bot VFX  - Associate Head of Creative Operations

Bot VFX

Chennai, Tamil Nadu, India (On-Site)
2 Months ago
SuperGaming - VFX Artist

SuperGaming

Pune, Maharashtra, India (On-Site)
2 Weeks ago
cyara - Software Engineer

cyara

Hyderabad, Telangana, India (Hybrid)
4 Months ago
Assystems - Manager / Senior Manager - Finance

Assystems

Chennai, Tamil Nadu, India (On-Site)
7 Months ago
Capgemini - Mechanical & Physical Engineer

Capgemini

Pune, Maharashtra, India (On-Site)
1 Week ago
NCR Atleos - Scrum Master

NCR Atleos

Hyderabad, Telangana, India (On-Site)
3 Weeks ago
Capgemini - Finance Controller

Capgemini

Pune, Maharashtra, India (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Category Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

About The Company

Gurugram, India (On-Site)

Gurugram, India (On-Site)

Gurugram, India (On-Site)

Gurugram, India (On-Site)

Gurugram, India (On-Site)

Gurugram, India (On-Site)

Gurugram, India (On-Site)

Gurugram, India (On-Site)

View All Jobs

Get notified when new jobs are added by pay2dc

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug