Senior Technical Program Manager, Gemini Evals

Snapshot

The role sits within Google DeepMind’s Gemini Evals TPM team. The team is responsible for designing and managing Gemini Evals suite to test model performance.

About us

Artificial Intelligence could be one of humanity’s most useful inventions. At Google DeepMind, we’re a team of scientists, engineers, machine learning experts and more, working together to advance the state of the art in artificial intelligence. We use our technologies for widespread public benefit and scientific discovery, and collaborate with others on critical challenges, ensuring safety and ethics are the highest priority.

The role

The Gemini Evals team is responsible for developing new ways to test and measure model performance. This is an open-ended problem that requires close collaboration between research and engineering and applying the state of the art AI/ML to conversational problems.

As a Technical Program Manager in the Gemini Eval team, you will be responsible for driving the development of new evals and the infrastructure to run them. This role requires a strong technical background in AI, excellent program management skills, and the ability to navigate the challenges of large-scale research and engineering efforts. You will work closely with research scientists, engineers, product managers, program managers and other cross-functional teams to solve problems around delivering and deploying massive models.

Key responsibilities

• Program Leadership & Strategy: Lead complex and ambiguous programs focusing on execution and delivery. Partner with research and engineering leads to translate strategic goals into well-defined program plans, timelines, and success metrics.
• Lead and drive progress and improvements in the Evals space to enable Gemini to increase its operating velocity.
• Manage dependencies across workstreams, functions and orgs.
• Drive and lead engineering teams to identify, prioritize and track tasks to completion targets.
• Identify risks and corresponding mitigation plans, and put into motion the solutions.
• Communicate progress, risks and plans to leadership on a regular basis.
• Identify operational and process improvements and quickly set-up lightweight processes to fill gaps.
• Manage multiple, time-sensitive projects simultaneously.
• Partner closely with PM and Eng leads to understand and influence product direction and ensure execution matches.

About you

In order to set you up for success as a Gemini Evals Team at Google DeepMind, we look for the following skills and experience:

• strong technical background, ideally a bachelor or master degree in computer science, data science or related fields
• strong biased towards action and looking for a start-up mindset and fast-paced environment and have an entrepreneurial drive
• Excellent technical understanding and communication ability, with the ability to distil sophisticated technical ideas to their essence
• Skilled at navigating, updating and defining complex processes
• strong and documented experience running large programs with extended teams of PM, Researchers, Engineers and UX-ers
• able to thrive in ambiguity and cross-team/cross-site collaborations
• Fluent in SQL and Python
• Experience with LLM and models evaluations

In addition, the following would be an advantage:

• Previous experience as data analyst, data scientist or software engineer is a plus

Note: In the event your application is successful and an offer of employment is made to you, any offer of employment will be conditional on the results of a background check, performed by a third party acting on our behalf. For more information on how we handle your data, please see our Applicant and Candidate Privacy Policy

At Google DeepMind, we value diversity of experience, knowledge, backgrounds and perspectives and harness these qualities to create extraordinary impact. We are committed to equal employment opportunity regardless of sex, race, religion or belief, ethnic or national origin, disability, age, citizenship, marital, domestic or civil partnership status, sexual orientation, gender identity, pregnancy, or related condition (including breastfeeding) or any other basis as protected by applicable law. If you have a disability or additional need that requires accommodation, please do not hesitate to let us know.