Senior Machine Learning Ops Engineer, ML System - Foundation Model

5 Months ago • 2 Years + • System Design • $184,300 PA - $337,250 PA

Job Summary

Job Description

ByteDance's Large Model Team seeks a Senior Machine Learning Ops Engineer to manage and optimize its large-scale ML systems. Responsibilities include ensuring efficient operation of systems for model development, training, and inference across multi-data center, multi-region, and multi-cloud environments. The role involves resource management, cost optimization, disaster recovery, and building monitoring tools. The ideal candidate possesses strong proficiency in Go/Python/Shell, Kubernetes, and containerization, along with experience in managing GPU servers and distributed systems. The position requires excellent analytical skills, documentation habits, and strong teamwork.
Must have:
  • Efficient ML system operation
  • Multi-cloud system stability
  • Resource management & cost optimization
  • Disaster recovery & cluster governance
  • Monitoring tool development
  • Proficiency in Go/Python/Shell, Kubernetes
Good to have:
  • Experience with large-scale ML distributed systems
  • GPU server operation and maintenance experience
Perks:
  • Medical, dental, and vision insurance
  • 401(k) savings plan with company match
  • Paid parental leave
  • Paid holidays and sick days
  • Paid Personal Time

Job Details

Responsibilities
About Bytedance Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content. Why Join Us Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. To us, every challenge, no matter how ambiguous, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always. At ByteDance, we create together and grow together. That's how we drive impact-for ourselves, our company, and the users we serve. Join us. Team Intro The ByteDance Large Model Team is committed to developing the most advanced AI large model technology in the industry, becoming a world-class research team, and contributing to technological and social development. The Large Model Team has a long-term vision and determination in the field of AI, with research directions covering NLP, CV, speech, and other areas. Relying on the abundant data and computing resources of the platform, the team has continued to invest in relevant fields and has launched its own general large model, providing multi-modal capabilities. The Machine Learning (ML) System sub-team combines system engineering and the art of machine learning to develop and maintain massively distributed ML training and Inference system/services around the world, providing high-performance, highly reliable, scalable systems for LLM/AIGC/AGI In our team, you'll have the opportunity to build the large scale heterogeneous system integrating with GPU/NPU/RDMA/Storage and keep it running stable and reliable, enrich your expertise in coding, performance analysis and distributed system, and be involved in the decision-making process. You'll also be part of a global team with members from the United States, China and Singapore working collaboratively towards unified project direction. Responsibilities: 1. Responsible for ensuring our ML systems are operating and running efficiently for large model development, training, evaluation, and inference 2. Responsible for the stability of offline tasks/services in multi-data center, multi-region, and multi-cloud scenarios 3. Responsible for resource management and planning, cost and budget, including computing and storage resources 4. Responsible for global system disaster recovery, cluster machine governance, stability of business services, resource utilisation improvement and operation efficiency improvement 5. Build software tools, products and systems to monitor and manage the ML infrastructure and services efficiently 6. Be part of the global team roster that ensures system and business on-call support
Qualifications
Minimum Qualifications: - Bachelor's degree or above, major in computer science, computer engineering or related - Strong proficiency in at least one programming language such as Go/Python/Shell in Linux environment - Strong hands-on experience with Kubernetes and containers skills, and have more than 2 years of relevant operation and maintenance experience - Possess excellent logical analysis ability, able to reasonably abstract and split business logic - Have good documentation principles and habits to be able to write and update workflow and technical documentation as required on time - Possess a strong sense of responsibility, good learning ability, communication ability and self-drive, good team spirit Preferred Qualifications - Engaged in the operation and maintenance of large-scale ML distributed systems - Experience in operation and maintenance of GPU servers ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too. ByteDance Inc. is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or a reasonable accommodation, please reach out to us at https://shorturl.at/cdpT2
Job Information
【For Pay Transparency】Compensation Description (Annually)

The base salary range for this position in the selected city is $184300 - $337250 annually.

Compensation may vary outside of this range depending on a number of factors, including a candidate’s qualifications, skills, competencies and experience, and location. Base pay is one part of the Total Package that is provided to compensate and recognize employees for their work, and this role may be eligible for additional discretionary bonuses/incentives, and restricted stock units.

Benefits may vary depending on the nature of employment and the country work location. Employees have day one access to medical, dental, and vision insurance, a 401(k) savings plan with company match, paid parental leave, short-term and long-term disability coverage, life insurance, wellbeing benefits, among others. Employees also receive 10 paid holidays per year, 10 paid sick days per year and 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure).

The Company reserves the right to modify or change these benefits programs at any time, with or without notice.

For Los Angeles County (unincorporated) Candidates:

Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state, and local laws including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Our company believes that criminal history may have a direct, adverse and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment:

1. Interacting and occasionally having unsupervised contact with internal/external clients and/or colleagues;

2. Appropriately handling and managing confidential information including proprietary and trade secret information and access to information technology systems; and

3. Exercising sound judgment.

Similar Jobs

Salesforce - Specialist Account Executive - Platform & Security (Commercial Accounts)

Salesforce

State Of São Paulo, Brazil (On-Site)
1 Month ago
Oliver Agency - Account Manager

Oliver Agency

Singapore (Remote)
1 Week ago
Justplay - Director of Data

Justplay

Berlin, Berlin, Germany (Remote)
2 Months ago
Capgemini - Products & Systems Engineer

Capgemini

Bengaluru, Karnataka, India (On-Site)
3 Weeks ago
160over90 - Account Manager, Talent Partnerships - Olympics

160over90

New York, United States (On-Site)
1 Month ago
Applied materials  - System Engineer

Applied materials

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Qualcomm - Senior Systems Performance Engineer

Qualcomm

San Diego, California, United States (On-Site)
1 Month ago
extreme network - STAFF SW SYSTEMS ENGINEER 9723 - WiFi Protocols-802.11

extreme network

Bengaluru, Karnataka, India (Hybrid)
2 Months ago
Roblox - Senior Software Engineer - Systems

Roblox

San Mateo, California, United States (On-Site)
2 Weeks ago
Ion - System Engineer Control Room, Italy

Ion

Parma, Emilia-Romagna, Italy (Hybrid)
8 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Monks - Associate Creative Director - Art (Contractual)

Monks

Noida, Uttar Pradesh, India (On-Site)
8 Months ago
HoYoverse - Senior Gameplay Programmer AI [USA]

HoYoverse

Santa Monica, California, United States (Remote)
1 Year ago
NVIDIA - Senior Site Reliability Engineer, HPC and LSF

NVIDIA

Durham, North Carolina, United States (On-Site)
3 Months ago
Rippling - Forward Deployed Engineer

Rippling

San Francisco, California, United States (On-Site)
2 Weeks ago
GoReel - Senior Game Mathematician

GoReel

Bratislava Region, Slovakia (Remote)
3 Months ago
Tencent - Senior Area Manager, WeChat Pay (Japan)

Tencent

(On-Site)
3 Months ago
Riot Games - Staff Software Engineer (UI)

Riot Games

Los Angeles, California, United States (On-Site)
2 Months ago
Tesla - Sales Advisor

Tesla

Erfurt, Thuringia, Germany (On-Site)
4 Months ago
Figma - Program Manager, Talent Enablement

Figma

San Francisco, California, United States (Remote)
2 Weeks ago
world resource institute - Consultant as Nature-based Solutions Economic Analyst

world resource institute

Padang, West Sumatra, Indonesia (On-Site)
2 Weeks ago

Get notifed when new similar jobs are uploaded

Jobs in Seattle, Washington, United States

bytedance - Student Researcher (Doubao (Seed) - Foundation Model - Generative AI)

bytedance

Seattle, Washington, United States (On-Site)
4 Months ago
HCL Tech - Python developer (data analysis)

HCL Tech

Texas, United States (On-Site)
1 Month ago
Zuora - Senior Product Manager, Billing

Zuora

Redwood City, California, United States (On-Site)
19 Hours ago
SBM Management - Custodial Lead

SBM Management

Monee, Illinois, United States (On-Site)
3 Months ago
Qualcomm - PMIC Reference HW Design Engineer

Qualcomm

San Diego, California, United States (On-Site)
1 Month ago
Dentsu - Senior Manager, Advanced Analytics/Insights

Dentsu

Detroit, Michigan, United States (Remote)
3 Weeks ago
bytedance - Research Scientist, Multimodality

bytedance

Seattle, Washington, United States (On-Site)
8 Months ago
Penn Interactive - Executive Host

Penn Interactive

Denver, Colorado, United States (Remote)
1 Month ago
Take-Two Interactive - Information Security Operations Analyst

Take-Two Interactive

Austin, Texas, United States (On-Site)
1 Month ago
Zones - Security Access Management Lead

Zones

United States (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

System Design Jobs

Apple - Sensing System HW Engineer

Apple

San Diego, California, United States (On-Site)
1 Month ago
Canonical - Senior Ubuntu Embedded IoT System Engineer

Canonical

Taipei City, Taiwan (On-Site)
1 Month ago
Visa - Chief Systems Architect

Visa

Auckland, Auckland, New Zealand (Hybrid)
6 Months ago
Sabre India - Sr. Software Systems Engineer

Sabre India

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Whatnot - Senior IT Systems & Ops Engineer

Whatnot

San Francisco, California, United States (On-Site)
2 Months ago
2K - Principal Systems Engineer

2K

Burnaby, British Columbia, Canada (Hybrid)
2 Months ago
Loft Orbital - Senior Satellite Systems Engineer

Loft Orbital

Abu Dhabi, Abu Dhabi, United Arab Emirates (On-Site)
1 Month ago
Alpha Sense - Sr. IT Engineer, Enterprise Systems Technology

Alpha Sense

Vancouver, British Columbia, Canada (Hybrid)
1 Month ago
bytedance - Software Engineer in Machine Learning Systems

bytedance

San Jose, California, United States (On-Site)
8 Months ago
Ion - System Engineer Jr, Italy

Ion

Italy (Hybrid)
8 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content.

San Jose, California, United States (On-Site)

San Jose, California, United States (On-Site)

Seattle, Washington, United States (On-Site)

Seattle, Washington, United States (On-Site)

San Jose, California, United States (On-Site)

View All Jobs

Get notified when new jobs are added by bytedance

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug