Intermediate Machine Learning Engineer, AI Framework

2 Months ago • All levels • Research Development

Job Summary

Job Description

As a Machine Learning Engineer on GitLab’s AI framework team, you'll contribute to shaping the future of AI-powered features. You will be responsible for merging evaluation tools, optimizing dataset management, and scaling validation infrastructure. You will work closely with other AI feature teams to ensure that every AI feature is robust and reliable. Responsibilities include designing and implementing technical evaluators for LLM assessment, building scalable evaluation pipelines, developing and managing datasets, collaborating with feature teams, and optimizing performance across ML evaluation systems. The role involves ensuring all solutions align with GitLab’s infrastructure and security protocols.
Must have:
  • Experience with LLM evaluation systems design.
  • Understanding of ML model architectures.
  • Expertise in ML evaluation metrics.
  • Ability to build production-grade ML infrastructure.
  • Experience with Python-based ML frameworks.
Good to have:
  • Familiarity with open-source development.
Perks:
  • Benefits to support your health, finances, and well-being
  • All remote, asynchronous work environment
  • Flexible Paid Time Off
  • Team Member Resource Groups
  • Equity Compensation & Employee Stock Purchase Plan
  • Growth and development budget
  • Parental leave
  • Home office support

Job Details

GitLab is an open core software company that develops the most comprehensive AI-powered DevSecOps Platform, used by more than 100,000 organizations. Our mission is to enable everyone to contribute to and co-create the software that powers our world. When everyone can contribute, consumers become contributors, significantly accelerating the rate of human progress. This mission is integral to our culture, influencing how we hire, build products, and lead our industry. We make this possible at GitLab by running our operations on our product and staying aligned with our values. Learn more about Life at GitLab.

Thanks to products like Duo Enterprise, and Duo Workflow, customers get the benefit of AI at every stage of the SDLC. The same principles built into our products are reflected in how our team works: we embrace AI as a core productivity multiplier. All team members are encouraged and expected to incorporate AI into their daily workflows to drive efficiency, innovation, and impact across our global organisation.

An overview of this role

Are you passionate about building robust frameworks to evaluate and ensure the reliability of AI models? As a Machine Learning Engineer on GitLab’s AI framework team, you’ll play a critical role in shaping the future of AI-powered features at GitLab. This is an exciting opportunity to work on impactful projects that directly influence the quality of GitLab’s AI capabilities.

You’ll help merge cutting-edge evaluation tools, optimize dataset management, and scale our validation infrastructure. Working closely with other AI feature teams, you’ll ensure that every AI feature we deliver is robust, reliable, and meets the highest quality standards.

Some challenges in this role include designing scalable solutions for LLM evaluation, consolidating disparate validation tools, and contributing to GitLab’s innovative AI roadmap.

Some examples of our projects: 

Consolidating Evaluation Tooling | The GitLab HandbookGitLab.org / AI Powered / ELI5

What You’ll Do  

  • Design and implement technical evaluators for LLM assessment.
  • Contribute to evaluation infrastructure consolidation efforts.
  • Build scalable evaluation pipelines and frameworks.
  • Develop and manage datasets and evaluation metrics.
  • Collaborate with feature teams to integrate validation solutions.
  • Optimize performance across ML evaluation systems.
  • Support improvements to GitLab’s AI-powered tools through validation.
  • Ensure all solutions align with GitLab’s infrastructure and security protocols.

What You’ll Bring 

  • Proven experience designing and implementing LLM evaluation systems.
  • Strong understanding of ML model architectures, including public vs. private implementations.
  • Expertise in ML evaluation metrics and dataset management.
  • Demonstrated ability to build production-grade ML infrastructure.
  • Practical experience with Python-based ML frameworks and evaluation tools (e.g., Langsmith, ELI5).
  • Excellent problem-solving skills with an engineering mindset.
  • Ability to collaborate in an asynchronous, remote-first environment.
  • Familiarity with open-source development and contribution is a plus.

About the team

The AIF team ensures that AI models across GitLab are reliable and well-validated. We focus on building robust evaluation frameworks, consolidating tools, and streamlining processes to scale validation efforts across GitLab’s AI infrastructure. Working on high-impact projects, the team partners with AI feature teams to deliver quality-focused solutions that enhance user trust and product performance.

How GitLab will support you

Please note that we welcome interest from candidates with varying levels of experience; many successful candidates do not meet every single requirement. Additionally, studies have shown that people from underrepresented groups are less likely to apply to a job unless they meet every single qualification. If you're excited about this role, please apply and allow our recruiters to assess your application.

 


Country Hiring Guidelines: GitLab hires new team members in countries around the world. All of our roles are remote, however some roles may carry specific location-based eligibility requirements. Our Talent Acquisition team can help answer any questions about location after starting the recruiting process.  

Privacy Policy: Please review our Recruitment Privacy Policy. Your privacy is important to us.

GitLab is proud to be an equal opportunity workplace and is an affirmative action employer. GitLab’s policies and practices relating to recruitment, employment, career development and advancement, promotion, and retirement are based solely on merit, regardless of race, color, religion, ancestry, sex (including pregnancy, lactation, sexual orientation, gender identity, or gender expression), national origin, age, citizenship, marital status, mental or physical disability, genetic information (including family medical history), discharge status from the military, protected veteran status (which includes disabled veterans, recently separated veterans, active duty wartime or campaign badge veterans, and Armed Forces service medal veterans), or any other basis protected by law. GitLab will not tolerate discrimination or harassment based on any of these characteristics. See also GitLab’s EEO Policy and EEO is the Law. If you have a disability or special need that requires accommodation, please let us know during the recruiting process.

Similar Jobs

Canonical - Desktop and Embedded Linux Field Engineer

Canonical

Taipei City, Taiwan (On-Site)
2 Months ago
Monzo - Senior Data Scientist

Monzo

London, England, United Kingdom (Remote)
2 Months ago
Inveniolsi - SAP BODS Consultant

Inveniolsi

Mumbai, Maharashtra, India (On-Site)
1 Month ago
Scientific Games - Senior Field Service Engineer

Scientific Games

Georgia, United States (Remote)
5 Months ago
Crytek - Senior Rendering Programmer

Crytek

Frankfurt Am Main, Hessen, Germany (On-Site)
2 Weeks ago
Bosch Group - Research & Development Engineer for MEMS technology

Bosch Group

Dresden, Saxony, Germany (On-Site)
1 Month ago
sound cloud - Senior Machine Learning Engineer

sound cloud

Berlin, Berlin, Germany (On-Site)
2 Months ago
Google - Technical Program Manager III, AI/ML, Cloud AI Systems

Google

Austin, Texas, United States (On-Site)
3 Months ago
Next Level Business Services - User Experience Researcher

Next Level Business Services

Santa Clara, California, United States (On-Site)
9 Months ago
bytedance - Research Scientist in Large Language Model, Graduates - Doubao (Seed) - 2024 Start (PhD)

bytedance

Seattle, Washington, United States (On-Site)
8 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Grammarly - Commercial (Mid-Market) Account Executive

Grammarly

United States (Hybrid)
1 Week ago
Tide - Senior Engineer, Flutter

Tide

Bulgaria (Remote)
3 Weeks ago
Aspire - Payment Operations Senior Manager

Aspire

Gurugram, India (Hybrid)
2 Months ago
CAE - Financial Analyst - Planning & Forecasting

CAE

Montreal, Quebec, Canada (On-Site)
1 Month ago
Resolver - Vice President Sales

Resolver

Toronto, Ontario, Canada (Hybrid)
1 Month ago
Cognite - Senior Implementation Project Manager

Cognite

Austin, Texas, United States (Hybrid)
2 Months ago
Flow - Paid Social Specialist

Flow

Miami, Florida, United States (On-Site)
2 Weeks ago
GT HQ - Network Engineer (NOC)

GT HQ

Kraków, Lesser Poland Voivodeship, Poland (Remote)
2 Weeks ago
Springer Group - Development Editor

Springer Group

Mexico City, Mexico (On-Site)
1 Week ago

Get notifed when new similar jobs are uploaded

Jobs in Canada

Barracuda - Cybersecurity Developer

Barracuda

Ottawa, Ontario, Canada (Hybrid)
4 Months ago
Airlab Inc  - Junior Programmer Artificial Intelligence

Airlab Inc

Quebec, Canada (On-Site)
4 Months ago
Rocket - Cloud Sales Account Executive (Mainframe Modernization)

Rocket

Ontario, Canada (Remote)
1 Month ago
AECOM - Senior Electrical Engineer - Buildings Projects

AECOM

Saskatoon, Saskatchewan, Canada (Hybrid)
1 Month ago
2K - Senior Operations Manager (Creative Dev)

2K

Montréal, Québec, Canada (Hybrid)
1 Month ago
People Can Fly - Senior Level Artist

People Can Fly

Montreal, Quebec, Canada (Remote)
2 Months ago
NetEase Games - Infrastructure Engineer

NetEase Games

Quebec, Canada (Remote)
4 Months ago
Larian Studios - Technical QA Tester Internship

Larian Studios

Quebec, Canada (On-Site)
5 Months ago
Salesforce - Enterprise Account Executive

Salesforce

Toronto, Ontario, Canada (On-Site)
2 Weeks ago
Lionbridge Games - VR Test Analyst

Lionbridge Games

Quebec, Canada (On-Site)
5 Months ago

Get notifed when new similar jobs are uploaded

Research Development Jobs

Tesla - EHS Technical Engineer R&D

Tesla

Baden-Württemberg, Germany (On-Site)
5 Months ago
Apple - AIML - Machine Learning Researcher, MLR

Apple

Cambridge, Massachusetts, United States (On-Site)
1 Month ago
Unity - Staff Machine Learning Engineer

Unity

San Francisco, California, United States (On-Site)
4 Weeks ago
Apple - Senior AI Application Engineer

Apple

Cupertino, California, United States (On-Site)
1 Month ago
Apple - AIML - Machine Learning Educator

Apple

New York, New York, United States (On-Site)
2 Months ago
Bazaar Voice - Machine Learning Engineer

Bazaar Voice

Belfast, Northern Ireland, United Kingdom (Hybrid)
1 Month ago
Eqvilent - Quantitative Researcher

Eqvilent

(Remote)
2 Months ago
Sword Health - Senior ML Engineer

Sword Health

Porto, Porto District, Portugal (Hybrid)
2 Years ago
C3 IoT - AI Engagement Manager / Director

C3 IoT

Redwood City, California, United States (On-Site)
2 Months ago
Ubisoft - Scientifique en données ML Senior _ Groupe Technologique Content Creation

Ubisoft

Montreal, Quebec, Canada (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded