Service Delivery – GenAI Monitoring & Analysis

Job Description

We are seeking a detail-oriented and analytical professional to join our Service Delivery team, focusing on Generative AI (GenAI) Monitoring and Analysis. In this role, you will oversee the performance, reliability, and user experience of deployed GenAI solutions. You will work across teams to identify performance trends, proactively detect issues, analyze usage and feedback data, and support the continuous improvement of AI services.

Key Responsibilities:

• Monitor the health and performance of deployed GenAI systems using logs, metrics, dashboards, and user feedback
• Analyze AI outputs, identify anomalies, and surface quality concerns or potential model drift
• Collaborate with engineering, product, and data science teams to triage and resolve issues
• Maintain documentation for monitoring processes, incident response, and known issues
• Track and report on KPIs related to GenAI usage, stability, responsiveness, and accuracy
• Drive root cause analysis and recommend operational improvements or retraining needs
• Facilitate feedback loops between end-users and technical teams to improve GenAI outputs
• Contribute to the development of playbooks, escalation paths, and runbooks for GenAI services
• Support readiness efforts for new GenAI releases and features from a service delivery perspective
• Remain flexible and shift focus to support other key-projects as business needs evolve

Required Qualifications:

• 3+ years in a service delivery, operations, data analysis, or product support role, ideally in a tech or AI environment
• Familiarity with Generative AI concepts and LLM (Large Language Model) applications
• Strong analytical skills with experience working with large datasets and interpreting system logs
• Experience using monitoring and incident management tools
• Effective communicator with a user-first mindset and the ability to work across functions
• Understanding of SLAs, KPIs, and service health monitoring best practices

Preferred Qualifications:

• Experience supporting or monitoring GenAI-powered applications in production environments
• Exposure to AI/ML model evaluation techniques or prompt testing workflows
• Knowledge of prompt engineering and/or fine-tuning LLMs
• Prior experience in an enterprise SaaS or platform environment

Basic Requirements

• Bachelor's degree
• 3+ years in a service delivery, operations, data analysis, or product support role, ideally in a tech or AI environment

Travel Requirements

Under 10%

Relocation Provided

None

Position Type

Experienced

Referral Payment Plan

Yes

Our U.S. Benefits include:

• Incentive Bonus Plans
• Medical, Dental, Vision benefits
• 401K
• 10 Paid Holidays
• Generous Paid Time Off Packages
• Employee Stock Purchase Plan
• Paid Parental & Family Leave
• and more!