Big Data Engineer - Data Lake / Feature Store

3 Months ago • All levels • Data Analysis

Job Summary

Job Description

The Big Data Engineer will be responsible for developing and optimizing the in-house Feature Store functionality based on Iceberg. They will also participate in optimizing the integration of Iceberg with various upper-level computing engines and involve in platform-related infrastructure development. The role requires working with Spark, Primus, and Ray to support offline data processing and distributed training for various business scenarios within the company. The team focuses on functional and performance optimizations for large-scale scenarios and the adoption of new-generation distributed application frameworks.

Job Details

About ByteDance Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content. Why Join Us Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. To us, every challenge, no matter how ambiguous, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. Join us. About The Team The batch processing team is responsible for the company's offline data processing and distributed training, supporting various business scenarios such as offline ETL and machine learning within the company. The components involved include the offline computing engine Spark, the in-house distributed training framework Primus, feature storage solutions like Iceberg and Hudi, as well as Ray, a next-generation distributed application framework. Faced with massive-scale scenarios, extensive functional and performance optimizations have been carried out in Spark, Primus, Feature Store, and support for the adoption of the new-generation distributed application framework Ray in relevant company scenarios. What you will be doing: - Responsible for the development and performance optimisation of the in-house Feature Store functionality based on Iceberg; - Participant in optimisation of the integration of Iceberg with various upper-level computing engines; - Involve in platform-related infrastructure development.

Similar Jobs

GoTo Group - Business Intelligence Analyst

GoTo Group

Jakarta, Jakarta, Indonesia (On-Site)
2 Months ago
FICO - Senior/Lead Research Engineer - AI/ML- Applied AI

FICO

United States (Remote)
1 Month ago
Unity - Sales Onboarding Intern

Unity

Bengaluru, Karnataka, India (On-Site)
2 Months ago
Nfocus solution - Test Engineer

Nfocus solution

Orlando, Florida, United States (On-Site)
1 Month ago
Optiv - Senior Analyst - SOC I

Optiv

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Varonis  - Applied Data Scientist

Varonis

Morrisville, North Carolina, United States (Hybrid)
2 Months ago
PayPal - Manager, Data Science

PayPal

Dublin, County Dublin, Ireland (Hybrid)
2 Months ago
Xsolla - Product Data Analyst

Xsolla

Baku, Azerbaijan (Hybrid)
3 Months ago
Capgemini - Risk and Finance Data Engineer

Capgemini

Pune, Maharashtra, India (On-Site)
3 Months ago
Applike - Data Science Lead (Playtime Team) (f/m/d)

Applike

Hamburg, Hamburg, Germany (Hybrid)
10 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Apple - AIML - Senior Engineering Program Manager, ML Lifecycle Platform

Apple

Cupertino, California, United States (On-Site)
1 Month ago
big red button - Senior Server Programmer

big red button

Los Angeles, California, United States (On-Site)
2 Months ago
ISS Stoxx - Software Development Lead

ISS Stoxx

Mumbai, Maharashtra, India (On-Site)
3 Months ago
Pokemon - Category Manager

Pokemon

Bellevue, Washington, United States (Hybrid)
1 Month ago
FlockSafety - Paid Social Analyst

FlockSafety

United States (Remote)
1 Month ago
Aptive - RTR Team Leader

Aptive

Suzhou, Jiangsu, China (On-Site)
3 Months ago
ShyftLabs - Senior Machine Learning Engineer

ShyftLabs

Toronto, Ontario, Canada (Hybrid)
3 Months ago
Cloud Imperium Games - Senior Software Engineer C# / .Net

Cloud Imperium Games

Manchester, England, United Kingdom (On-Site)
2 Months ago
Accenture - Clinical Data Svs Associate

Accenture

Bengaluru, Karnataka, India (On-Site)
4 Months ago
The Walt Disney Company - Senior Effects Technical Director

The Walt Disney Company

Sydney, New South Wales, Australia (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in Singapore

bytedance - Security Operation Engineer, Security Assurance

bytedance

Singapore (On-Site)
6 Months ago
Nagarro - Associate Director, Delivery

Nagarro

Singapore (On-Site)
9 Months ago
Aspire - MLRO

Aspire

Singapore (Hybrid)
3 Months ago
Sony Pictures Entertainment - Commercial Strategy Manager, Television Distribution, Asia

Sony Pictures Entertainment

Singapore (On-Site)
2 Months ago
Marvell - Senior Staff Product Engineer

Marvell

Singapore (On-Site)
1 Year ago
bytedance - Cloud Solutions Technical Account Manager

bytedance

Singapore (On-Site)
4 Months ago
OKX - HRBP Director

OKX

Singapore (On-Site)
4 Weeks ago
OKX - Senior Data Product Manager

OKX

Singapore (On-Site)
3 Months ago
Marsh McLennan - Fiduciary Operations Lead - Asia Placement Hub

Marsh McLennan

Singapore (Hybrid)
1 Year ago
bytedance - Product Solutions Architect - Enterprise Security

bytedance

Singapore (On-Site)
9 Months ago

Get notifed when new similar jobs are uploaded

Data Analysis Jobs

velotio technologies  - Senior Engineer (Data Engineer)

velotio technologies

Maharashtra, India (Remote)
4 Months ago
BetterMe - Business Analyst (Web)

BetterMe

Kyiv, Kyiv City, Ukraine (On-Site)
3 Months ago
Next Level Business Services - Oracle Functional Business Analyst

Next Level Business Services

San Francisco, California, United States (On-Site)
10 Months ago
Aeries technology - Business Process Analyst

Aeries technology

Bengaluru, Karnataka, India (On-Site)
3 Months ago
Apple - Sr Data Scientist, Measurement

Apple

Cupertino, California, United States (On-Site)
2 Months ago
CRB workforce  - Data Engineer

CRB workforce

Seattle, Washington, United States (On-Site)
2 Months ago
Western Digital - Intern - Data Science

Western Digital

Phra Nakhon Si Ayutthaya, Thailand (On-Site)
1 Month ago
Jam City - Senior Data Analyst

Jam City

Toronto, Ontario, Canada (On-Site)
1 Year ago
Cubic corporation - Market Data Services Analyst

Cubic corporation

Warsaw, Masovian Voivodeship, Poland (Hybrid)
1 Year ago
Cygames - Data Engineer / Analytics Platform Development / Tokyo

Cygames

Tokyo, Japan (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content.
View All Jobs

Get notified when new jobs are added by bytedance

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug