LLM Performance Operation, Analysts (Safety)

5 Months ago • All levels • Monetization

Job Summary

Job Description

As an LLM Performance Operations Analyst (Safety) at ByteDance, you'll be a core member of the LLM Global Data Team. Responsibilities include measuring and analyzing LLM performance across various dimensions, creating metrics, designing and conducting tests, and delivering performance analyses to identify issues in LLM training and product design. You'll collaborate with product managers and engineers to build testing workflows and feedback loops, and work closely with the annotation team to assess training data quality. The role involves exposure to potentially harmful content, requiring strong critical thinking and analytical skills, and excellent written communication in English. Experience in consulting, market research, or NLP model evaluation is preferred. A paper test is required before interviews.
Must have:
  • Professional proficiency in English
  • Critical thinking and analytical skills
  • Ability to meet deadlines and generate creative solutions
  • Experience in LLM performance measurement
  • Collaboration with product and engineering teams
Good to have:
  • Consulting, market research, or auditing experience
  • NLP model and search product performance evaluation experience
  • Deep interest in LLMs

Job Details

Responsibilities
About ByteDance Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok, Helo, and Resso, as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content. Why Join Us Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. To us, every challenge, no matter how ambiguous, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. Join us. As a core member of our LLM Global Data Team, you'll be at the heart of our operations. Gain first-hand experience in understanding how to measure and analyze performance of LLM on various dimensions. Your Role Will Involve: 1. Gaining direct experience in LLM and LLM-relevant product performance measurement, including creating metrics, designing and conducting tests, to help tech and product teams to understand LLM and product performance. 2. Delivering performance and feedback analysis to identify issues in LLM training and product design. 3. Collaborating with product managers and engineers to build testing workflows, create feedback loops, and enhance the effectiveness of testing processes and standards. 4. Working closely with the annotation team to gauge the quality of training data sets.
Qualifications
Minimum Qualifications: 1. Professional proficiency in English with excellent writing and critique skills. 2. Critical thinking and analytical skills are required. 3. Comfort with meeting deadlines and generating creative solutions, including the use of technology and tools, to enhance the quality of both individual and team outputs. Preferred Qualifications: 1. Experience in consulting, market research, auditing, operational process design and improvement is valued. Experience as NLP model and search product performance evaluators is strongly preferred. 2. A deep interest in LLM is essential. The ideal candidate is an enthusiastic learner who finds engagement with diverse case studies stimulating. Note:This role requires a paper test prior to interviews. It is possible that this role will be exposed to harmful content as part of the core role/as part of project/by chance. This may occur in the form of text, images, videos related to every-day life, but it can also include (but is not limited to) bullying; hate speech; child safety; depictions of harm to self and others, and harm to animals. ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.

Similar Jobs

Netflix - Manager, Live Action Content Licensing - Japan

Netflix

Tokyo, Japan (On-Site)
6 Months ago
Every matrix - Operations Deputy Manager

Every matrix

Zagreb, Croatia (Hybrid)
1 Month ago
PwC - Senior Associate - Consulting - Workforce Transformation (Talent Pool)

PwC

Jakarta, Jakarta, Indonesia (On-Site)
7 Months ago
PwC - Senior Associate IT Assurance Services

PwC

Makati, Metro Manila, Philippines (On-Site)
7 Months ago
ByteDance - Data Management and Strategy Intern

ByteDance

Taguig, Metro Manila, Philippines (On-Site)
1 Month ago
InMobiInMobi - Lead - Growth - 1Weather

InMobiInMobi

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Epic Games - Director, Product Management (Subscription + Pass Products)

Epic Games

Cary, North Carolina, United States (On-Site)
3 Months ago
ByteDance - Creator Operations Intern (Philippines)

ByteDance

Taguig, Metro Manila, Philippines (On-Site)
3 Months ago
ByteDance - Financial Risk Strategy Expert - Global Payment

ByteDance

Bangkok, Bangkok, Thailand (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

PwC - Senior Associate -SAP ABAP-Kolkata-TC

PwC

Kolkata, West Bengal, India (On-Site)
7 Months ago
Keywords Studios - AI Prompt & Language Specialist (Italian)

Keywords Studios

Montreal, Quebec, Canada (Remote)
1 Month ago
NVIDIA - Senior Manager, Data Quality Operations

NVIDIA

Santa Clara, California, United States (On-Site)
1 Month ago
PwC - Advisory - Deals - Forensics - Data Analyst - Associate 2 - Kolkata

PwC

Kolkata, West Bengal, India (On-Site)
4 Months ago
PwC - Deals - Consultant, Forensic Services - Subcontractor Opportunity

PwC

Amman, Amman Governorate, Jordan (On-Site)
6 Months ago
Netflix - Metadata & Taxonomy Program Manager

Netflix

United States (Remote)
2 Months ago
Probably Monsters - Design Director

Probably Monsters

Bellevue, Washington, United States (On-Site)
5 Months ago
Ubisoft - Development Tester

Ubisoft

Nova Scotia, Canada (On-Site)
1 Month ago
Experis - VR AI Video Games Engineer

Experis

Portland, Oregon, United States (On-Site)
9 Months ago

Get notifed when new similar jobs are uploaded

Jobs in Singapore

Bushiroad - Development Executive

Bushiroad

Singapore, Singapore (On-Site)
2 Months ago
ByteDance - Threat Intelligence Engineer, Security Assurance - 2025 Start

ByteDance

Singapore (On-Site)
5 Months ago
Ubisoft - Lead Data Analyst

Ubisoft

Singapore (On-Site)
2 Months ago
ByteDance - Backend Engineer, Video-On-Demand - 2025 Start

ByteDance

Singapore (On-Site)
6 Months ago
The Walt Disney Company - Marketing Ops Consultant - Contract

The Walt Disney Company

Singapore, Singapore (On-Site)
5 Months ago
ByteDance - Backend Software Engineer - Foundational Technology

ByteDance

Singapore (On-Site)
1 Month ago
ByteDance - Seed - LLM Performance Operation Analyst (Non-safety)

ByteDance

Singapore (On-Site)
4 Months ago
ByteDance - LLM Training Operation (Reasoning and Knowledge) - Specialist

ByteDance

Singapore (On-Site)
6 Months ago
ByteDance - Solution Architect (Edge Cloud)

ByteDance

Singapore (On-Site)
2 Months ago
ByteDance - Backend Software Engineer (Business Infra), ByteCloud - 2025 Start

ByteDance

Singapore (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Monetization Jobs

ByteDance - Product Leader - Product Strategy - Singapore

ByteDance

Singapore (On-Site)
6 Months ago
ByteDance - Livestream Agency Manager - Philippines

ByteDance

Taguig, Metro Manila, Philippines (On-Site)
4 Months ago
ByteDance - AI Voice Technical Support Engineer (Global Market)

ByteDance

Singapore (On-Site)
1 Month ago
ByteDance - Content Insights Analyst, Lifestyle

ByteDance

Los Angeles, California, United States (On-Site)
1 Month ago
ByteDance - Payment Strategy Intern - Global Payment

ByteDance

Singapore (On-Site)
1 Month ago
PAPAYA - Monetization Manager - Solitaire

PAPAYA

Tel Aviv-Yafo, Tel Aviv District, Israel (On-Site)
1 Month ago
ByteDance - E-commerce Agency Partnership Intern (Philippines) - 2025 Start

ByteDance

Taguig, Metro Manila, Philippines (On-Site)
1 Month ago
Moon Active - Product Monetization Manager

Moon Active

Tel Aviv-Yafo, Tel Aviv District, Israel (On-Site)
1 Month ago
ByteDance - Ad Delivery Algorithm Intern - Game

ByteDance

Singapore (On-Site)
2 Months ago
ByteDance - Frontend Software Engineer Intern (PDI-CSP-FE-i18n )- 2025 Summer (BS/MS)

ByteDance

Seattle, Washington, United States (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Where imagination meets innovation, delivering limitless gaming experiences.

San Diego, California, United States (On-Site)

San Jose, California, United States (On-Site)

Dubai, Dubai, United Arab Emirates (On-Site)

New York, New York, United States (On-Site)

San Jose, California, United States (On-Site)

San Jose, California, United States (On-Site)

Seattle, Washington, United States (On-Site)

View All Jobs

Get notified when new jobs are added by ByteDance

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug