Software Engineer, LLM Inference Scheduling Intern

About the Team ByteDance Doubao Large Model Team was established in 2023, dedicated to developing the most advanced AI large model technology in the industry, becoming a world-class research team, and contributing to the development of technology and society. The Doubao large model team has a long-term vision and determination in the field of AI, with research directions covering NLP, CV, speech, etc. They have laboratories and research positions in China, Singapore, the US and other places. The team relies on sufficient data, computing and other resources on the platform, continuously invests in related fields, and has launched self-developed general large models, providing MultiModal Machine Learning capabilities. Downstream support includes 50 + businesses such as Doubao, Coze, Dreamina, and is open to enterprise customers through Volcengine. Currently, Doubao APP has become the largest AIGC application in the Chinese market. 1. Engage in the scheduling and orchestration of heterogeneous resources for LLM, including computing power pooling, elastic resource mixed deployment, tidal resource borrowing, and quota management. 2. Participate in the multi-role, multi-stage scheduling of PD and EP scenarios for LLM inference services, as well as KVCache-centric scheduling, in order to achieve dynamic, timely, and accurate scaling in and out for services 3. Participate in the optimal scheduling of computing resources, RDMA high-speed network resources, and cache/storage resources via technical means, and fully utilize the computing power of large-scale distributed clusters 4. Contribute to the stability of LLM services and accomplish problem localization, diagnosis, isolation, and rapid recovery across diverse heterogeneous resources (GPU, CPU, NPU, etc), multi-cloud environments, and various network traffic scenarios via online and offline multi-system linkage 5. Engage in task/service scheduling within multi-datacenter, multi-region, and multi-cloud scenarios to ensure the rational distribution of load Qualifications Minimum Qualifications: 1. Demonstrate proficiency in at least 1 to 2 languages, such as C++, Go, Python, and Shell, within the Linux environment. 2. Possess an understanding of Kubernetes architecture and ecosystem, and be familiar with container technologies including Docker, Containerd, Kata, and Podman. 3. Exhibit excellent logical analysis skills, with the ability to reasonably abstract and break down business logic. 4. Have a strong sense of work responsibility, good learning capabilities, communication skills, and self-motivation, and be capable of rapid response and action. 5. Maintain good habits of creating working documents, and write and update workflow and technical documents in a timely manner as required. Preferred Qualifications: 1. Familiar with at least one mainstream machine learning framework or inference engine and have relevant experience in optimizing the inference performance of large models. 2. Have practical experience in resource scheduling and service orchestration of large models, and have participated in the design, development, and maintenance of large-scale distributed systems. 3. Have an understanding of GPU systems and architectures. 4. Have experience in publishing papers at top academic conferences in the field of computer systems (including but not limited to OSDI, NSDI, SOSP, FAST, MLSYS, Eurosys). Candidates can apply to a maximum of two positions and will be considered for jobs in the order you apply. The application limit is applicable to ByteDance and its affiliates' jobs globally. Applications will be reviewed on a rolling basis - we encourage you to apply early. Successful candidates must be able to commit to at least 3 months long internship period. By submitting an application for this role, you accept and agree to our global applicant privacy policy, which may be accessed here: https://jobs.bytedance.com/en/legal/privacy. If you have any questions, please reach out to us at apac-earlycareers@bytedance.com Job Information About Doubao (Seed) Founded in 2023, the ByteDance Doubao (Seed) Team, is dedicated to pioneering advanced AI foundation models. Our goal is to lead in cutting-edge research and drive technological and societal advancements.​ With a strong commitment to AI, our research areas span deep learning, reinforcement learning, Language, Vision, Audio, AI Infra and AI Safety. Our team has labs and research positions across China, Singapore, and the US.​ Why Join ByteDance Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life - a mission we work towards every day.​ As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Join us.​ Diversity & Inclusion​ ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.​