Lead Machine Learning Engineer - Applied Scientist

Upwork ($UPWK) is the world’s largest work marketplace, connecting businesses with highly skilled professionals worldwide. From entrepreneurs to Fortune 100 enterprises, companies trust Upwork’s platform to access expert talent, leverage AI-powered work solutions, and drive meaningful business outcomes.

Upwork’s AI-powered platform has facilitated over $20 billion in economic opportunity for professionals worldwide. With professionals spanning 10,000+ skills, including AI and machine learning, software development, sales and marketing, customer support, finance and accounting, and more, Upwork empowers businesses of all sizes to scale, innovate, and build agile teams.

We’re looking for a Lead Machine Learning Engineer / Applied Scientist with a passion for rigorously evaluating and improving the performance of LLMs and AI agents. In this role, you will focus on building feedback loops, defining success metrics, and driving measurable improvements to the quality and reliability of our intelligent systems. This is a rare opportunity to shape how evaluation and iteration are embedded in the product lifecycle for AI at scale.

You’ll partner closely with research, engineering, and product teams to embed your insights into Upwork’s AI infrastructure. This includes designing testbeds for agentic workflows, refining prompts and orchestration strategies, and guiding the iteration of ML-powered features to deliver better outcomes for our users. Your work will directly influence the success of our most advanced AI initiatives.

Responsibilities

• Develop and own evaluation pipelines for agentic LLM systems, enabling consistent measurement across simulation, benchmark, and live-user scenarios.
• Define and iterate on quality metrics that guide the training, tuning, and deployment of LLMs and agents.
• Lead experiments to assess and improve system behaviors across dimensions such as correctness, safety, latency, and helpfulness.
• Collaborate with cross-functional partners to integrate insights from evaluation into product development and deployment pipelines.
• Build automated testing and monitoring tools that scale with the complexity of agent behaviors and LLM responses.
• Share findings and improvements through documentation, dashboards, and internal demos, contributing to a culture of continuous learning and excellence.

What it takes to catch our eye

• Deep familiarity with evaluation methodologies for LLMs or autonomous agents, including benchmark selection, prompt sensitivity analysis, and human-in-the-loop review processes.
• Hands-on experience in Python and ML frameworks such as PyTorch, with the ability to analyze outputs and drive performance iteration.
• Proven ability to work across teams and disciplines to align ML evaluation work with product and business goals.
  Comfortable operating in ambiguous problem spaces and taking initiative to define structure, priorities, and impact.
• Passion for continuous improvement, strong documentation habits, and a collaborative, inclusive working style.

Come change how the world works.

At Upwork, you’ll shape the future of work for a global, remote-first workforce, creating economic opportunities for professionals worldwide. While we have a physical office in Palo Alto, we currently hire full-time employees in 21 U.S. states, making it easier than ever to join our mission from wherever you call home.

Our culture is built on trust, risk-taking, customer focus, and excellence, all in service of our core mission: to create economic opportunities so people have better lives. We embrace authenticity and inclusion, encouraging everyone to bring their whole selves to work. Personal and professional growth is a priority here, supported through development programs, mentorship, and our Upwork Belonging Communities.

We’re proud to offer benefits that go beyond the basics, including comprehensive medical coverage for you and your family, unlimited PTO, a 401(k) plan with matching, 12 weeks of paid parental leave, and an Employee Stock Purchase Plan. Visit our Life at Upwork page to learn more about our values, working principles, and the overall employee experience.

Ready to help shape the future of work? Check out our Careers page and follow us on LinkedIn, Facebook, Instagram, TikTok, and X to learn more about life at Upwork.

Upwork is an Equal Opportunity Employer committed to recruiting and retaining a diverse and inclusive workforce. We do not discriminate based on race, religion, color, national origin, gender (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, or other legally protected characteristics under federal, state, or local law.

Please note that a criminal background check may be required once a conditional job offer is made. Qualified applicants with arrest or conviction records will be considered in accordance with applicable law, including the California Fair Chance Act and local Fair Chance ordinances.

To learn more about how Upwork processes and protects your personal information as part of the application process, please review our Global Job Applicant Privacy Notice