Big Data Engineer - Data Lake / Feature Store

1 Month ago • All levels • Data Analysis

Job Summary

Job Description

The Big Data Engineer will be responsible for developing and optimizing the in-house Feature Store functionality based on Iceberg. They will also participate in optimizing the integration of Iceberg with various upper-level computing engines and involve in platform-related infrastructure development. The role requires working with Spark, Primus, and Ray to support offline data processing and distributed training for various business scenarios within the company. The team focuses on functional and performance optimizations for large-scale scenarios and the adoption of new-generation distributed application frameworks.

Job Details

About ByteDance Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content. Why Join Us Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. To us, every challenge, no matter how ambiguous, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. Join us. About The Team The batch processing team is responsible for the company's offline data processing and distributed training, supporting various business scenarios such as offline ETL and machine learning within the company. The components involved include the offline computing engine Spark, the in-house distributed training framework Primus, feature storage solutions like Iceberg and Hudi, as well as Ray, a next-generation distributed application framework. Faced with massive-scale scenarios, extensive functional and performance optimizations have been carried out in Spark, Primus, Feature Store, and support for the adoption of the new-generation distributed application framework Ray in relevant company scenarios. What you will be doing: - Responsible for the development and performance optimisation of the in-house Feature Store functionality based on Iceberg; - Participant in optimisation of the integration of Iceberg with various upper-level computing engines; - Involve in platform-related infrastructure development.

Similar Jobs

Tesla - Business Operations Analyst

Tesla

Mumbai, Maharashtra, India (On-Site)
4 Months ago
Google - Software Engineer III, Google Cloud Platforms

Google

San Francisco, California, United States (On-Site)
1 Month ago
Nice - Services Sales Manager (EMEA)

Nice

London, England, United Kingdom (Hybrid)
5 Days ago
Homa Games - Game Tech Data Analyst

Homa Games

Paris, Île-de-France, France (On-Site)
1 Month ago
Epic Games - Vendor Manager, Trust & Safety

Epic Games

United States (On-Site)
2 Months ago
The Walt Disney Company - Business Systems Analyst

The Walt Disney Company

Hong Kong (On-Site)
2 Months ago
Scorewarrior - BI Developer & Data Engineer (Marketing)

Scorewarrior

Limassol, Limassol, Cyprus (On-Site)
1 Month ago
Synechron - Business Analyst – Fidessa or Ullink (Securities Lending / Stock Loan)

Synechron

Jersey City, New Jersey, United States (On-Site)
2 Weeks ago
Nintendo - Business Systems Analyst (Salesforce Project Manager)

Nintendo

Redmond, Washington, United States (Hybrid)
4 Months ago
HCL Tech - Reporting & Data Analyst

HCL Tech

Texas, United States (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

OKX - Chatbot Specialist

OKX

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)
8 Months ago
PwC - Senior Data Scientist

PwC

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)
8 Months ago
Logrhytm - Program Manager - Partner Performance & Retention

Logrhytm

Pune, Maharashtra, India (On-Site)
1 Day ago
Next Level Business Services - Splunk Developer

Next Level Business Services

San Diego, California, United States (On-Site)
8 Months ago
Rive - Security & DevOps Engineer

Rive

San Francisco, California, United States (Hybrid)
2 Weeks ago
QS Quacquarelli Symonds  - Global Sales Operations and Enablement Manager

QS Quacquarelli Symonds

London, England, United Kingdom (Hybrid)
2 Months ago
Alten Technology - Manufacturing Engineer (Composites)

Alten Technology

Lansing, Michigan, United States (On-Site)
1 Week ago
FICO - Senior C++ Engineer (Parallel Processing, Low Latency)

FICO

United States (Remote)
1 Month ago
Marsh McLennan - Graduate Analyst, Employee Engagement and Experience

Marsh McLennan

Melbourne, Victoria, Australia (Hybrid)
3 Weeks ago

Get notifed when new similar jobs are uploaded

Jobs in Singapore

Riot Games - Director, HRBP - APAC/China

Riot Games

Singapore (On-Site)
1 Month ago
Alpha Sense - Enterprise Account Executive, Corporate

Alpha Sense

Singapore, Singapore (On-Site)
6 Months ago
Razer - Senior Category Specialist

Razer

Singapore (On-Site)
3 Weeks ago
bytedance - Senior Software Engineer, Anti-DDoS

bytedance

Singapore (On-Site)
5 Months ago
Workato - Intern, Product Management

Workato

Singapore (On-Site)
1 Month ago
Nervos Foundation - Blockchain Developer Relations & Ecosystem Development

Nervos Foundation

Singapore (Remote)
2 Years ago
Sony Music Career - Business Development Manager/Sr. Manager South East Asia (Talent Pooling)

Sony Music Career

Singapore (On-Site)
1 Month ago
bytedance - Global Monetization Product Counsel, Ads

bytedance

Singapore (On-Site)
2 Months ago
Ubisoft - Senior Rewards Partner

Ubisoft

Singapore, Singapore (On-Site)
6 Months ago
Qualcomm - Capacity Planning Engineer

Qualcomm

Singapore (On-Site)
4 Weeks ago

Get notifed when new similar jobs are uploaded

Data Analysis Jobs

Nagarro - Associate Principal Engineer, Big Data

Nagarro

Sri Lanka (Remote)
8 Months ago
Scientific Games - Senior Business Analyst

Scientific Games

Georgia, United States (Remote)
2 Months ago
Tide - Lead Data Scientist

Tide

Delhi, India (Hybrid)
1 Month ago
Extreme Inc. - Data Analysis Engineer

Extreme Inc.

Tokyo, Japan (Hybrid)
1 Month ago
Codeway Studios - Data Analyst

Codeway Studios

Istanbul, İstanbul, Türkiye (On-Site)
1 Month ago
dun bradstreet - Data Operations Analyst

dun bradstreet

Jacksonville, Florida, United States (On-Site)
3 Days ago
Brillio - AWS Data Architect/Senior Lead Data Engineer

Brillio

Gurugram, Haryana, India (Hybrid)
3 Days ago
Capgemini - Data Business Analyst

Capgemini

Pune, Maharashtra, India (On-Site)
2 Weeks ago
Zazz - Data Engineer (6–8 Years) Adhoc

Zazz

India (On-Site)
6 Months ago
Apple - AMP Fraud Data Analyst

Apple

Austin, Texas, United States (On-Site)
3 Weeks ago

Get notifed when new similar jobs are uploaded

About The Company

Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content.

San Jose, California, United States (On-Site)

San Jose, California, United States (On-Site)

Seattle, Washington, United States (On-Site)

Atlanta, Georgia, United States (On-Site)

Seattle, Washington, United States (On-Site)

View All Jobs

Get notified when new jobs are added by bytedance

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug