Big Data Engineer, Data Lake / Feature Store

5 Months ago • 2 Years + • Monetization

Job Summary

Job Description

The batch processing team at ByteDance is responsible for offline data processing and distributed training. You will be developing and optimizing the in-house Feature Store functionality based on Iceberg, participating in optimizing the integration of Iceberg with various upper-level computing engines, and being involved in platform-related infrastructure development.
Must have:
  • Bachelor's Degree or above in Computer Science or related fields
  • 2+ years of relevant development experience
  • Strong programming ability in Java, Python, C++
  • Experience with large-scale distributed systems
  • In-depth knowledge of data lake formats like Delta, Hudi, or Iceberg
Good to have:
  • In-depth research or practical experience in Hadoop, Spark, Flink, Presto
  • Experience with open-source big data computing frameworks

Job Details

Responsibilities
About ByteDance Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content. Why Join Us Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. To us, every challenge, no matter how ambiguous, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. Join us. About The Team The batch processing team is responsible for the company's offline data processing and distributed training, supporting various business scenarios such as offline ETL and machine learning within the company. The components involved include the offline computing engine Spark, the in-house distributed training framework Primus, feature storage solutions like Iceberg and Hudi, as well as Ray, a next-generation distributed application framework. Faced with massive-scale scenarios, extensive functional and performance optimizations have been carried out in Spark, Primus, Feature Store, and support for the adoption of the new-generation distributed application framework Ray in relevant company scenarios. What you will be doing: - Responsible for the development and performance optimisation of the in-house Feature Store functionality based on Iceberg; - Participant in optimisation of the integration of Iceberg with various upper-level computing engines; - Involve in platform-related infrastructure development.
Qualifications
Minimum Qualifications - Bachelor's Degree or above, majoring in Computer Science, or related fields, with 2+ years of relevant development experience in the field with a strong programming ability, and proficiency in Java, Python, C++, with the ability to develop and optimize large-scale distributed systems. - In-depth research and relevant experience in one or more data lake formats such as Delta, Hudi, or Iceberg. Preferred Qualifications - In-depth research or practical experience in open-source big data computing frameworks and scenarios like Hadoop, Spark, Flink, Presto, and more. ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and bring joy. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too. #LI-CT

Similar Jobs

Bigpoint - Lead Game Developer

Bigpoint

Hamburg, Hamburg, Germany (Remote)
3 Months ago
Patreon - Senior Fullstack Software Engineer, Payments

Patreon

San Francisco, California, United States (Hybrid)
4 Weeks ago
Matific - Software Engineer

Matific

Sydney, New South Wales, Australia (On-Site)
3 Weeks ago
PlayStation Global - Sr. ML Software Engineer

PlayStation Global

United States (Remote)
4 Weeks ago
Egnyte - Staff Software Engineer

Egnyte

Mountain View, California, United States (Hybrid)
5 Months ago
ByteDance - Senior Manager, Product Management - Customer Service Platform - International E-commerce

ByteDance

Seattle, Washington, United States (On-Site)
4 Weeks ago
ByteDance - Research Scientist, Code Generation

ByteDance

Seattle, Washington, United States (On-Site)
5 Months ago
ByteDance - Central Strategy and Sales Policy Associate - Monetization Strategy & Operations

ByteDance

New York, New York, United States (On-Site)
1 Month ago
Tap Nation - Senior User Acquisition Manager

Tap Nation

(Remote)
5 Months ago
ByteDance - Machine Learning Engineer - Model Serving Infrastructure

ByteDance

Seattle, Washington, United States (On-Site)
4 Weeks ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

ByteDance - Software Development Engineer in Test

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
Google - Staff Software Engineer, Machine Learning

Google

Mountain View, California, United States (On-Site)
1 Week ago
ByteDance - Senior Software Engineer - Serverless Compute Infrastructure

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
Inworld AI - Staff Backend Engineer

Inworld AI

Mountain View, California, United States (Hybrid)
2 Days ago
Zeta - Software Development Engineer _ II Backend

Zeta

Bengaluru, Karnataka, India (On-Site)
6 Months ago
JustPlay - Backend Engineer

JustPlay

Berlin, Berlin, Germany (Hybrid)
4 Weeks ago
ByteDance - Software Developer Graduate (Routing Verification & Emulation)

ByteDance

San Jose, California, United States (On-Site)
1 Month ago
Sporty Group - Backend Engineer

Sporty Group

(Remote)
9 Months ago
The Walt Disney Company - Lead IT Developer

The Walt Disney Company

Montévrain, Île-de-France, France (On-Site)
4 Days ago
Dream Games - Senior Software Engineer

Dream Games

İstanbul, Türkiye (On-Site)
10 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Singapore

ByteDance - Cash Management FX Expert, Treasury - Global Payment

ByteDance

Singapore (On-Site)
2 Days ago
ByteDance - Lark APAC Integrated Marketing Intern

ByteDance

Singapore (On-Site)
3 Months ago
Google - Database Sales Specialist Manager, Google Cloud

Google

Singapore (On-Site)
1 Week ago
Aspire - Director of Client Treasury

Aspire

Singapore (On-Site)
6 Months ago
ByteDance - Regional Head of Solution Architect, Cloud Security

ByteDance

Singapore (On-Site)
2 Months ago
ByteDance - Senior Privacy Security Product Manager - Information System

ByteDance

Singapore (On-Site)
3 Months ago
ByteDance - Principal Site Reliability Engineer, CDN

ByteDance

Singapore (On-Site)
6 Months ago
Riot Games - Product Coordinator, VALORANT SEA (Contract)

Riot Games

Singapore (On-Site)
2 Months ago
Razer - Solutions Architect

Razer

Singapore (On-Site)
6 Months ago
Riot Games - Technical Artist II - League of Legends, Seasons (Contract)

Riot Games

Singapore (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

Monetization Jobs

Sawhorse Productions - Mobile Game Designer - Monetization

Sawhorse Productions

California, United States (Remote)
6 Hours ago
ByteDance - Graduate Account Management, Beauty (Philippines E-Commerce)

ByteDance

Taguig, Metro Manila, Philippines (On-Site)
1 Month ago
ByteDance - Strategy Intern, BytePlus

ByteDance

Singapore (On-Site)
1 Week ago
ByteDance - Machine Learning Engineer - AML Algorithm

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
ByteDance - Financial Risk Strategy Expert - Global Payment

ByteDance

Singapore (On-Site)
5 Months ago
Netflix - Technical Program Manager (L6), Ads Measurement

Netflix

Los Gatos, California, United States (On-Site)
6 Days ago
Jam City - Monetization Manager - Mobile Gaming Industry

Jam City

Toronto, Ontario, Canada (On-Site)
9 Months ago
ByteDance - Senior Natural Language Processing Algorithm Engineer

ByteDance

Seattle, Washington, United States (On-Site)
1 Month ago
ByteDance - Music Partnerships Manager - SoundOn

ByteDance

Bangkok, Bangkok, Thailand (On-Site)
4 Months ago
ByteDance - Video Codec Algorithm Engineer - Multimedia Lab

ByteDance

Seattle, Washington, United States (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Where imagination meets innovation, delivering limitless gaming experiences.

Dublin, County Dublin, Ireland (On-Site)

London, England, United Kingdom (On-Site)

Bangkok, Bangkok, Thailand (On-Site)

San Jose, California, United States (On-Site)

State Of São Paulo, Brazil (On-Site)

San Jose, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by ByteDance

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug