UI/AI Engineer

Job Summary

We are building first party 1P Tunable Agents for specific tasks and scenarios. As an AI Engineer on this team, you will design and build agentic AI experiences that make these 1P agents shine, serving as the primary entry point for Copilot Tuning. You will collaborate across UX, product, and platform to prototype, harden, and ship multi-turn, goal-oriented agents that learn from feedback and continuously improve.

Duties/Responsibilities

• Improve Agent UX end to end from prompt flows and state management to error handling and recovery
• Use LLMs and tool use patterns to drive agentic orchestration and multi turn task completion
• Design and implement mechanisms to collect user feedback explicit and implicit and iterate quickly based on insights
• Prototype evaluate and productionization agent behaviors policies and guardrails aligned with safety privacy and compliance standards
• Instrument agents with telemetry define success metrics and run AB experiments to optimize quality and performance
• Integrate with Copilot Tuning systems so that agents serve as the primary entry point for tuning workflows and controls
• Collaborate closely with design PM and partner engineering teams to land cohesive delightful experiences
• Document designs write clean and testable code and contribute to shared components and best practices
• Other related duties as assigned

Required Skills/Abilities

• Hands on experience building with large language models LLMs promptresponse patterns and agent frameworks
• Experience with React and frontend development
• Proficiency in Python or TypeScriptC with strong software engineering fundamentals testing CICD reliability
• Experience designing multi turn conversational or task oriented experiences familiarity with UX heuristics
• Knowledge of orchestration patterns tool calling function calling retrieval planning and state management for agents
• Ability to define and monitor quality metrics task success latency grounding guardrail triggers and run experiments
• Familiarity with feedback systems ratings rubrics trace collection and techniques to learn from user signals
• Understanding of responsible AI principles content safety data privacy and enterprise compliance requirements
• Experience working with cloud services eg Azure and vectorretrieval systems for grounding
• Strong cross functional collaboration and communication skills