Seed - LLM Performance Operation Analyst (Non-safety)

1 Hour ago • All levels • Data Analyst

About the job

Job Description

As an LLM Performance Operation Analyst at ByteDance's Doubao (Seed) team, you'll measure and analyze LLM performance across various dimensions. Responsibilities include creating metrics, designing and conducting tests, delivering performance analysis to identify issues in LLM training and product design, collaborating with product teams to build testing workflows and enhance testing processes, and working with the annotation team to assess training data quality. This role requires strong analytical, project management, and communication skills, with experience in LLM performance evaluation preferred.
Must have:
  • Professional proficiency in English
  • Critical thinking & analytical skills
  • Strong project management capabilities
  • LLM performance measurement experience
  • Collaboration with engineers & product managers
Good to have:
  • Consulting, market research, or auditing experience
  • NLP model and search product performance evaluation experience
  • Deep interest in LLMs
Not hearing back from companies?
Unlock the secrets to a successful job application and accelerate your journey to your next opportunity.
Responsibilities
About Doubao (Seed) Founded in 2023, the ByteDance Doubao (Seed) Team, is dedicated to pioneering advanced AI foundation models. Our goal is to lead in cutting-edge research and drive technological and societal advancements. With a strong commitment to AI, our research areas span deep learning, reinforcement learning, Language, Vision, Audio, AI Infra and AI Safety. Our team has labs and research positions across China, Singapore, and the US. Leveraging substantial data and computing resources and through continued investment in these domains, we have developed a proprietary general-purpose model with multimodal capabilities. In the Chinese market, Doubao models power over 50 ByteDance apps and business lines, including Doubao, Coze, and Dreamina, and is available to external enterprise clients via Volcano Engine. Today, the Doubao app stands as the most widely used AIGC application in China. Why Join Us Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. To us, every challenge, no matter how ambiguous, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always. At ByteDance, we create together and grow together. That's how we drive impact-for ourselves, our company, and the users we serve. Join us. As a core member of our LLM Global Data Team, you'll be at the heart of our operations. Gain first-hand experience in understanding how to measure and analyze performance of LLM on various dimensions. Your Role Will Involve: 1. Gaining direct experience in LLM and LLM-relevant product performance measurement, including creating metrics & designing and conducting tests, to help tech and product teams to understand LLM and product performance. 2. Delivering performance and feedback analysis to identify issues in LLM training and product design. 3. Collaborating with product managers and engineers to build testing workflows, create feedback loops, and enhance the effectiveness of testing processes and standards. 4. Working closely with the annotation team to gauge the quality of training data sets.
Qualifications
Minimum Qualifications: 1. Professional proficiency in English with excellent writing and critique skills. 2. Critical thinking, numerate and analytical skills are required 3. Strong project management capabilities with proven ability to work effectively with both internal and external stakeholders; comfortable handling ambiguous requirements and adapting to changes while maintaining project momentum 4. Comfortable with meeting deadlines and generating creative solutions, including the use of technology and tools, to enhance the quality of both individual and team outputs Preferred Qualifications: 1. Experience in consulting, market research, auditing, operational process design and improvement is valued. Experience as NLP model and search product performance evaluation is strongly preferred. 2. A deep interest in LLM is essential. The ideal candidate is an enthusiastic learner who finds engagement with diverse case studies stimulating. ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.
View Full Job Description

Add your resume

80%

Upload your resume, increase your shortlisting chances by 80%

About The Company

Where imagination meets innovation, delivering limitless gaming experiences.

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)

San Jose, California, United States (On-Site)

San Jose, California, United States (On-Site)

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)

View All Jobs

Get notified when new jobs are added by ByteDance

Similar Jobs

Scale AI - QA Engineer, Generative AI

Scale AI, Argentina (On-Site)

Krafton  - [AI] AI Engineer - NLP/Chatbot (3년 이상)

Krafton , South Korea (On-Site)

ByteDance - Software Engineer, Architecture and Infrastructure

ByteDance, United States (On-Site)

ByteDance - Research Scientist, Reinforcement Learning

ByteDance, United States (On-Site)

Duolingo - Senior Data Scientist

Duolingo, United States (On-Site)

Gallagher - Data Scientist

Gallagher, India (On-Site)

Dream Game Studios - SDE 3 - ML & Data Platform

Dream Game Studios, India (On-Site)

GoTo Group - Geospatial Analyst

GoTo Group, Indonesia (On-Site)

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Spyke Games - Data Scientist

Spyke Games, Türkiye (On-Site)

Balbix - Senior Staff/Principal SDET AI

Balbix, India (On-Site)

Hitachi - Data Science

Hitachi, India (On-Site)

Netflix - Machine Learning Engineer Intern, Summer 2025

Netflix, United States (On-Site)

ASSIST Software - Other Positions

ASSIST Software, (Remote)

JustPlay - Senior Data Scientist (all genders)

JustPlay, Germany (Hybrid)

Get notifed when new similar jobs are uploaded

Jobs in Singapore

PwC - Tax Admin - Resource Planner

PwC, Singapore (On-Site)

Aspire - Product Director, FinCrime Risk

Aspire, Singapore (On-Site)

Razer - Solutions Architect

Razer, Singapore (On-Site)

ByteDance - Organization Management Advisor - 2025 Start

ByteDance, Singapore (On-Site)

NinjaVan - HR Operations Associate

NinjaVan, Singapore (On-Site)

IGG - Senior Backend Engineer

IGG, Singapore (On-Site)

PwC - Oracle EPM Associate

PwC, Singapore (On-Site)

Sinch - Chief of Staff, APAC

Sinch, Singapore (Hybrid)

Get notifed when new similar jobs are uploaded

Data Analyst Jobs

Logitech - Program Data Analyst

Logitech, China (Hybrid)

Nielsen Holdings - Senior Data Scientist

Nielsen Holdings, India (Hybrid)

Gallagher - Data Scientist

Gallagher, India (On-Site)

Sports Interactive - Business Intelligence Analyst

Sports Interactive, United Kingdom (Hybrid)

The Walt Disney Company - Data Governance Engineer

The Walt Disney Company, United States (On-Site)

Match Group - Senior Data Scientist (Product Analytics)

Match Group, United States (Hybrid)

Sabre India - Data Scientist

Sabre India, India (On-Site)

Get notifed when new similar jobs are uploaded