Service Delivery – GenAI Monitoring & Analysis

This role focuses on monitoring and analyzing Generative AI solutions within the Service Delivery team. Responsibilities include overseeing performance, reliability, and user experience of GenAI systems, identifying trends, detecting issues, analyzing usage data, and supporting continuous improvement. The position involves collaboration with engineering, product, and data science teams to resolve issues and enhance AI services, ensuring optimal operation and user satisfaction.

Must Have

• Monitor GenAI system health and performance using logs, metrics, dashboards, and user feedback.
• Analyze AI outputs, identify anomalies, and surface quality concerns or model drift.
• Collaborate with engineering, product, and data science teams to triage and resolve issues.
• Track and report on KPIs related to GenAI usage, stability, responsiveness, and accuracy.
• Drive root cause analysis and recommend operational improvements or retraining needs.
• Facilitate feedback loops between end-users and technical teams to improve GenAI outputs.
• 3+ years in service delivery, operations, data analysis, or product support, ideally in tech/AI.
• Familiarity with Generative AI concepts and LLM applications.
• Strong analytical skills with experience working with large datasets and interpreting system logs.
• Experience using monitoring and incident management tools.
• Effective communication and cross-functional collaboration skills.
• Understanding of SLAs, KPIs, and service health monitoring best practices.

Good to Have

• Experience supporting or monitoring GenAI-powered applications in production environments.
• Exposure to AI/ML model evaluation techniques or prompt testing workflows.
• Knowledge of prompt engineering and/or fine-tuning LLMs.
• Prior experience in an enterprise SaaS or platform environment.

Perks & Benefits

• Incentive Bonus Plans
• Medical benefits
• Dental benefits
• Vision benefits
• 401K with Company Match
• 10 Paid Holidays
• Generous Paid Time Off Packages
• Employee Stock Purchase Plan
• Paid Parental & Family Leave

Department Overview

A smarter store starts with smarter communication. Theatro, our user-friendly, voice-driven AI platform provides retail workers with real-time, discreet, and efficient communication — helping them stay safe and productive while delivering top-notch customer service.

Job Description

We are seeking a detail-oriented and analytical professional to join our Service Delivery team, focusing on Generative AI (GenAI) Monitoring and Analysis. In this role, you will oversee the performance, reliability, and user experience of deployed GenAI solutions. You will work across teams to identify performance trends, proactively detect issues, analyze usage and feedback data, and support the continuous improvement of AI services.

Key Responsibilities:

• Monitor the health and performance of deployed GenAI systems using logs, metrics, dashboards, and user feedback
• Analyze AI outputs, identify anomalies, and surface quality concerns or potential model drift
• Collaborate with engineering, product, and data science teams to triage and resolve issues
• Maintain documentation for monitoring processes, incident response, and known issues
• Track and report on KPIs related to GenAI usage, stability, responsiveness, and accuracy
• Drive root cause analysis and recommend operational improvements or retraining needs
• Facilitate feedback loops between end-users and technical teams to improve GenAI outputs
• Contribute to the development of playbooks, escalation paths, and runbooks for GenAI services
• Support readiness efforts for new GenAI releases and features from a service delivery perspective
• Remain flexible and shift focus to support other key-projects as business needs evolve

Required Qualifications:

• 3+ years in a service delivery, operations, data analysis, or product support role, ideally in a tech or AI environment
• Familiarity with Generative AI concepts and LLM (Large Language Model) applications
• Strong analytical skills with experience working with large datasets and interpreting system logs
• Experience using monitoring and incident management tools
• Effective communicator with a user-first mindset and the ability to work across functions
• Understanding of SLAs, KPIs, and service health monitoring best practices

Preferred Qualifications:

• Experience supporting or monitoring GenAI-powered applications in production environments
• Exposure to AI/ML model evaluation techniques or prompt testing workflows
• Knowledge of prompt engineering and/or fine-tuning LLMs
• Prior experience in an enterprise SaaS or platform environment