LLM Global Data - LLM Performance Operation Analyst

1 Month ago • All levels • Monetization

Job Summary

Job Description

As an LLM Global Data - LLM Performance Operation Analyst at ByteDance, you'll measure and analyze the performance of Large Language Models (LLMs) across various dimensions. Responsibilities include creating metrics, designing and conducting tests to understand LLM and product performance, delivering performance analysis to identify issues in LLM training and product design, collaborating with product teams to build testing workflows and feedback loops, and working with the annotation team to assess training data quality. The role requires strong analytical, project management, and communication skills, with experience in NLP model and search product performance evaluation being highly preferred. A deep interest in LLMs and a passion for creative problem-solving are essential.
Must have:
  • Professional English proficiency
  • Critical thinking & analytical skills
  • Strong project management
  • LLM performance measurement expertise
  • Collaboration with tech & product teams
Good to have:
  • Consulting/market research experience
  • NLP model & search product evaluation experience
  • Deep interest in LLMs

Job Details

Responsibilities
About ByteDance Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content. Why Join Us Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. To us, every challenge, no matter how ambiguous, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always. At ByteDance, we create together and grow together. That's how we drive impact-for ourselves, our company, and the users we serve. Join us. As a core member of our LLM Global Data Team, you'll be at the heart of our operations. Gain first-hand experience in understanding how to measure and analyze performance of LLM on various dimensions. Your Role Will Involve: 1. Gaining direct experience in LLM and LLM-relevant product performance measurement, including creating metrics & designing and conducting tests, to help tech and product teams to understand LLM and product performance. 2. Delivering performance and feedback analysis to identify issues in LLM training and product design. 3. Collaborating with product managers and engineers to build testing workflows, create feedback loops, and enhance the effectiveness of testing processes and standards. 4. Working closely with the annotation team to gauge the quality of training data sets.
Qualifications
Minimum Qualifications: 1. Professional proficiency in English with excellent writing and critique skills. 2. Critical thinking, numerate and analytical skills are required 3. Strong project management capabilities with proven ability to work effectively with both internal and external stakeholders; comfortable handling ambiguous requirements and adapting to changes while maintaining project momentum 4. Comfortable with meeting deadlines and generating creative solutions, including the use of technology and tools, to enhance the quality of both individual and team outputs Preferred Qualifications: 1. Experience in consulting, market research, auditing, operational process design and improvement is valued. Experience as NLP model and search product performance evaluation is strongly preferred. 2. A deep interest in LLM is essential. The ideal candidate is an enthusiastic learner who finds engagement with diverse case studies stimulating. ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.

Similar Jobs

Plarium - Director of Gen-AI

Plarium

Herzliya, Tel Aviv District, Israel (On-Site)
2 Months ago
Visa - Visa Consulting and Analytics Director

Visa

Tbilisi, Tbilisi, Georgia (On-Site)
6 Months ago
Tencent - Product Lead, Game Marketing Technology - SaaS Platform

Tencent

Los Angeles, California, United States (On-Site)
1 Month ago
WebMD - Sr. Product Manager, Clinical Interoperability

WebMD

Newark, New Jersey, United States (On-Site)
3 Months ago
ByteDance - Algorithm Intern (Video Codec - Realtime Codec Optimizations - Multimedia Streaming) - 2025 Summer (PhD)

ByteDance

San Diego, California, United States (On-Site)
4 Months ago
ByteDance - Mobile App Engineering Intern (Product RD and Infrastructure - Global E- Commerce) - 2025 Summer (BS/MS)

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
ByteDance - Contingent Workforce Management Expert

ByteDance

Singapore (On-Site)
1 Week ago
InMobiInMobi - Senior Analyst - Revenue / Growth Analytics

InMobiInMobi

Bengaluru, Karnataka, India (On-Site)
3 Days ago
ByteDance - Ad Delivery Algorithm Engineer-Game

ByteDance

Singapore (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Granicus - SLED Account Executive - Special Districts

Granicus

United States (Remote)
6 Months ago
PlayStation Global - Market Research Analyst

PlayStation Global

London, England, United Kingdom (On-Site)
2 Weeks ago
Google - Creative Development Lead, Large Customer Sales

Google

Tokyo, Japan (On-Site)
6 Days ago
Saviynt - Sr. Director (Application Access Governance) -  Governance Risk & Compliance

Saviynt

El Segundo, California, United States (Hybrid)
6 Months ago
Saviynt - Lead Product Manager - Integration Framework

Saviynt

Bengaluru, Karnataka, India (Hybrid)
6 Months ago
NVIDIA - Senior Boot Reset Silicon Hardware Engineer

NVIDIA

Santa Clara, California, United States (Hybrid)
3 Months ago
ZeptoLab - Game Economy Designer

ZeptoLab

(Remote)
2 Weeks ago
Riot Games - Senior Researcher, Wild Rift

Riot Games

Shanghai, Shanghai, China (On-Site)
1 Month ago
Google - Senior New Business Account Executive

Google

New York, New York, United States (On-Site)
1 Week ago
ByteDance - Benefits Business Partner

ByteDance

Seattle, Washington, United States (On-Site)
1 Week ago

Get notifed when new similar jobs are uploaded

Jobs in Singapore

ByteDance - Data Analyst - Global Payment

ByteDance

Singapore (On-Site)
4 Weeks ago
ByteDance - Product Manager, Payment Solutions - Global Payment

ByteDance

Singapore (On-Site)
6 Months ago
ByteDance - Data Analyst - Global Payment

ByteDance

Singapore (On-Site)
1 Week ago
ByteDance - Edge Network Engineer

ByteDance

Singapore (On-Site)
1 Week ago
The Walt Disney Company - Marketing, Media & VIP Guest Services Manager

The Walt Disney Company

Singapore, Singapore (On-Site)
1 Month ago
Animoca Brands - Investment and Strategic Partnership Associate/Manager

Animoca Brands

Singapore, Singapore (Hybrid)
1 Month ago

Get notifed when new similar jobs are uploaded

Monetization Jobs

Xsolla - Customer Support Representative

Xsolla

Baku, Azerbaijan (On-Site)
2 Months ago
ByteDance - Mobile App Engineering Intern (Product RD and Infrastructure - Global E- Commerce) - 2025 Summer (BS/MS)

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
ByteDance - Senior Site Reliability Engineer - Applied Machine Learning

ByteDance

San Jose, California, United States (On-Site)
4 Weeks ago
ByteDance - Machine Learning Engineer

ByteDance

San Jose, California, United States (On-Site)
2 Months ago
ByteDance - BNPL Product Manager -Global Payment

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
ByteDance - Video Pipeline Engineer/Architect- Pico- San Jose

ByteDance

San Jose, California, United States (On-Site)
4 Months ago
Voodoo - Growth Strategy and Operations Manager

Voodoo

Paris, Île-de-France, France (On-Site)
2 Weeks ago
ByteDance - Research Scientist Intern (Doubao (Seed) - Music Foundation Model) - 2024 Summer (PhD)

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
ByteDance - Tech Lead Machine Learning Engineer

ByteDance

Seattle, Washington, United States (On-Site)
1 Week ago
ByteDance - Research Engineer in Large Model System

ByteDance

San Jose, California, United States (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Where imagination meets innovation, delivering limitless gaming experiences.

Dublin, County Dublin, Ireland (On-Site)

London, England, United Kingdom (On-Site)

Bangkok, Bangkok, Thailand (On-Site)

San Jose, California, United States (On-Site)

State Of São Paulo, Brazil (On-Site)

San Jose, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by ByteDance

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug