Research Scientist/Engineer - Multimodal Interaction & World Model

5 Months ago • All levels • Artificial Intelligence • Research & Development

Job Summary

Job Description

The Research Scientist/Engineer will contribute to the Multimodal Interaction & World Model team, focusing on cutting-edge research in multimodal intelligence and virtual reality interaction. Responsibilities include exploring multimodal understanding, generative models, reinforcement learning, and AIGC; optimizing large-scale multimodal models; developing multimodal agents for virtual worlds; and using pre-training and simulation to model various environments. The role requires expertise in areas such as computer vision, machine learning, and AI, with a strong emphasis on building human-level multimodal understanding and interaction capabilities. The successful candidate will contribute to advancing the exploration and development of multimodal assistant products.
Must have:
  • Research in multimodal understanding, generation, and AI.
  • Large-scale model optimization and data construction.
  • Multimodal agent development for virtual worlds.
  • Experience with pre-training and simulation techniques.
Good to have:
  • Publications in top conferences (CVPR, ECCV, ICCV, NeurIPS, ICLR, SIGGRAPH).
  • Proficiency in C/C++ or Python.
  • Experience leading influential projects in related fields.

Job Details

Responsibilities
Established in 2023, the ByteDance Doubao (Seed) Team is dedicated to building industry-leading AI foundation models. We aim to do world-leading research and foster both technological and social progress. With a long-term vision and a strong commitment to the AI field, the Team conducts research in a range of areas including natural language processing (NLP), computer vision (CV), and speech recognition and generation. It has labs and researcher roles in China, Singapore, and the US. Leveraging substantial data and computing resources and through continued investment in these domains, our team has built a proprietary general-purpose model with multimodal capabilities. In the market, Doubao models power over 50 ByteDance apps and business lines, including Doubao, Coze, and Dreamina, and was launched to external enterprise clients through Volcano Engine. The Doubao app is the most used AIGC app in China. Why Join Us Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. To us, every challenge, no matter how ambiguous, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. Join us. About the team Welcome to the Multimodal Interaction & World Model team. Our mission is to solve the challenge of multimodal intelligence、virtual reality world interaction in AI. We conduct cutting-edge research on areas such as Foundations and applications of multimodal understanding models, Multimodal agent and inference, Unified models for generation and understanding, World Model. Our team is comprising experienced research scientists and engineers who are dedicated to developing models that boast human-level multimodal understanding and interaction capabilities. The team also aspires to advance the exploration and development of multimodal assistant products. We foster a feedback-driven environment to continuously enhance our foundation technologies. Come join us in shaping the future of AI and transforming the product experience for users worldwide. Responsibilities - Explore and research multi-modal understanding, generative, machine learning, reinforcement learning, AIGC, computer vision, artificial intelligence and other cutting-edge technologies. - Explore the basic model of large-scale/ultra-large-scale multi-modal understanding and generation interweaving, and carry out extreme system optimization; Data construction, instruction fine-tuning, preference alignment, model optimization; Improve the ability of data synthesis, scalable oversight, model reasoning and planning, build a comprehensive, objective and accurate evaluation system, and explore and improve the ability of large models. - Explore and break through the advanced capabilities of multi-modal models and world models, including but not limited to multi-modal RAG, visual COT and Agent, and build a universal multi-modal Agent for GUI/ games and other virtual worlds. - Use pre-training, simulation and other technologies to model various environments in the virtual/real world, provide the basic ability of multi-modal interactive exploration, promote application landing, and develop new technologies and new products with artificial intelligence technology as the core.
Qualifications
Minimum Qualifications: - Bachelor degree or above, computer, electronics, mathematics and other related majors. - Have in-depth research in one or more fields such as computer vision, multimodal, AIGC, machine learning, rendering generation, etc.. - Excellent analytical and problem solving skills; Ability to solve large model training and application problems; Ability to explore solutions independently. - With good communication and collaboration skills, proactive work, able to harmoniously cooperate with the team to explore new technologies and promote technological progress. Preferred Qualifications: - With excellent basic algorithms, solid foundation of machine learning, familiar with CV, AIGC, NLP, RL, ML and other fields of technology, CVPR, ECCV, ICCV, NeurIPS, ICLR, SIGGRAPH or SIGGRAPH Asia and other top conferences/journals published papers are preferred. - With excellent coding ability, proficient in C/C++ or Python programming language, ACM/ICPC, NOI/IOl, Top Coder, Kaggle and other competition winners preferred. - In the fields of multimodal, large model, basic model, world model, RL, rendering generation, leading projects with too much influence is preferred. ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.

Similar Jobs

NVIDIA - Senior Math Libraries Engineer - Dense Linear Algebra

NVIDIA

California, United States (Hybrid)
3 Months ago
Google - Software Engineer III, Android, Pixel Telephony

Google

Bengaluru, Karnataka, India (On-Site)
4 Days ago
Google - Senior Staff Software Engineer, Infrastructure, Google Cloud

Google

Bengaluru, Karnataka, India (On-Site)
4 Days ago
Google - Senior Software Engineer, Infrastructure

Google

Sunnyvale, California, United States (On-Site)
4 Days ago
Google - Senior Software Engineer, Embedded Systems/Firmware, Quantum

Google

Mountain View, California, United States (On-Site)
4 Days ago
Ness Digital - AI/ML Engineer

Ness Digital

Iași, Iași County, Romania (Remote)
3 Weeks ago
Google - Software Engineering Manager, TPU Systems, Platforms Infrastructure

Google

Taipei City, Taiwan (On-Site)
3 Days ago
Google - Software Engineer III, AI/ML, Google Photos

Google

Bengaluru, Karnataka, India (On-Site)
4 Days ago
Meta - Research Scientist Intern, Language and Multimodal Research for MetaAI (PhD)

Meta

Bellevue, Washington, United States (On-Site)
5 Months ago
Haptic - Senior AI/ML Engineer

Haptic

Paris, Île-de-France, France (Remote)
3 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

NVIDIA - Senior ASIC Verification Engineer - GPU Memory Subsystem

NVIDIA

Durham, North Carolina, United States (On-Site)
3 Weeks ago
Google - Technical Lead, System Test

Google

Sunnyvale, California, United States (On-Site)
4 Days ago
Omnissa - Staff Data Scientist

Omnissa

Bengaluru, Karnataka, India (Hybrid)
7 Months ago
NVIDIA - Senior Software Engineer - Conversational AI

NVIDIA

Pune, Maharashtra, India (On-Site)
1 Month ago
Google - Software Engineer III, Android

Google

Bucharest, Bucharest, Romania (On-Site)
1 Day ago
Tencent - Research Intern (NLP)

Tencent

California, United States (On-Site)
1 Month ago
ByteDance - Senior Backend Software Engineer - Customer Service Platform

ByteDance

Seattle, Washington, United States (On-Site)
1 Week ago
ByteDance - Senior Research Scientist- Foundation Model, Vision and Language

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
Google - Software Engineer, Video Stack, ChromeOS

Google

Taipei City, Taiwan (On-Site)
4 Days ago
Microsoft - Senior X++ Developer

Microsoft

Bengaluru, Karnataka, India (On-Site)
1 Day ago

Get notifed when new similar jobs are uploaded

Jobs in Singapore

ByteDance - Ethics and Compliance Specialist (Training and Communications)

ByteDance

Singapore (On-Site)
1 Week ago
Salesforce - 2025 PhD Intern - AI Research, Singapore

Salesforce

Singapore, Singapore (On-Site)
6 Months ago
Salesforce - Sales Development Representative, Thailand Market

Salesforce

Singapore, Singapore (On-Site)
3 Weeks ago
ByteDance - DevOps Engineer, Applied Machine Learning Engine - 2025 Start

ByteDance

Singapore (On-Site)
5 Months ago
Alphasense - Enterprise Account Executive, Corporate

Alphasense

Singapore, Singapore (On-Site)
4 Months ago
GoTo Group - Lead Software Engineer - Cartography

GoTo Group

Singapore (On-Site)
5 Months ago
Google - Partner Operations Manager, YouTube

Google

Singapore (On-Site)
3 Days ago
ByteDance - Global Network Commercial and Business Development Manager

ByteDance

Singapore (On-Site)
5 Months ago
Google - Sales Manager, Pursuit, Data Analytics, Google Cloud

Google

Singapore (On-Site)
4 Days ago
ByteDance - HR System Solution Expert - HR Operations

ByteDance

Singapore (On-Site)
4 Months ago

Get notifed when new similar jobs are uploaded

Artificial Intelligence Jobs

ByteDance - Senior Research Scientist, Foundation Model, Speech Understanding

ByteDance

San Jose, California, United States (On-Site)
5 Months ago
Google - Software Engineer III, Core Machine Learning, Google Cloud

Google

Sunnyvale, California, United States (On-Site)
5 Months ago
Google - Technical Program Manager III, NPI System Software, Cloud AI Systems

Google

Sunnyvale, California, United States (On-Site)
4 Days ago
Google - Software Engineer, Compiler Frontend, Silicon

Google

Mountain View, California, United States (On-Site)
3 Days ago
Zoox - Senior/Staff Software Engineer - Prediction Integration

Zoox

Foster City, California, United States (Hybrid)
6 Months ago
ION - Software/Artificial Intelligence Engineer - Graduate Development Program

ION

Pisa, Tuscany, Italy (Hybrid)
3 Months ago
NVIDIA - Senior Software Engineer - Automated Parallel Programming

NVIDIA

Santa Clara, California, United States (Remote)
3 Months ago
Google - Software Engineer III, AI/ML, Geo

Google

Mountain View, California, United States (On-Site)
4 Days ago
Google - Senior Software Engineer, LLM Code Agents

Google

Mountain View, California, United States (On-Site)
4 Days ago
Krafton  - Applied Research Scientist/Engineer - LLM Game Agent

Krafton

Seoul, South Korea (On-Site)
3 Weeks ago

Get notifed when new similar jobs are uploaded

About The Company

Where imagination meets innovation, delivering limitless gaming experiences.

San Jose, California, United States (On-Site)

San Jose, California, United States (Hybrid)

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)

Taguig, Metro Manila, Philippines (Hybrid)

Los Angeles, California, United States (On-Site)

New York, New York, United States (On-Site)

Pasig, Metro Manila, Philippines (On-Site)

View All Jobs

Get notified when new jobs are added by ByteDance

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug