LLM Quality Analyst

WHO WE ARE: MagicSchool is the premier generative AI platform for teachers. We're just over 2 years old, and more than 6 million teachers from all over the world have joined our platform. Join a top team at a fast growing company that is working towards real social impact. Make an account and try us out at our website and connect with our passionate community on our Wall of Love.

Role Description

As a LLM Quality Analyst, you'll work with the Trust, Safety and Quality team to support the quality and reliability of MagicSchool's AI outputs. This is an excellent early career role who wants to work in tech and explore different career paths like product management or data analysis. You will work closely with the Senior LLM Quality Analyst, handling feedback intake, ground truthing LLM responses, and helping maintain evaluation test suites. You will be organized, self-motivated, and excited to get things done quickly in a fast-paced startup environment.

What You'll Do

• Manage LLM Feedback Intake and Triage
• Intake, triage, replicate, and prioritize user feedback on LLM outputs
• Identify patterns in quality issues and escalate appropriately
• Act as human-in-the-loop for quality checks
• Manage and develop test cases
• Review and label LLM-generated responses for evaluators
• Generate ground truth datasets to validate evaluator accuracy and consistency
• Support Evaluator and Judge Development
• Help test and validate new judges as they are developed
• Draft and iterate on judge prompts and scoring rubrics to improve coverage and quality
• Assist with prompt adjustments and writing
• Build and Maintain Evaluation Test Suites
• Create and maintain test suites that allow for regular evaluation of LLM performance and evaluator rubrics
• Compile collections of prompts, expected responses, and target evaluation scores
• Ensure test coverage includes edge cases and diverse content
• Support Evaluation Dashboards
• Assist in building dashboards to visualize and monitor evaluator outputs, consistency, and regressions
• Collect and prepare data for analysis

Qualifications/Competencies/Skills

• Gets a lot done: Works hard, resourceful, does whatever it takes
• Self-starter: Takes initiative, doesn't wait to be told what to do
• Organized and detail-oriented: Can manage multiple workstreams and maintain clear documentation
• Adaptable: Smart, learns fast, curious
• Builds relationships easily: Emotionally intelligent, warm communicator
• Strong communication skills: Team-first mindset, highly collaborative
• Comfortable working in spreadsheets and organizing data
• Basic analytical skills and ability to identify patterns in data
• Strong written communication skills for drafting prompts and documentation
• Experience with LLMs or AI products

Experience

• Education background - worked in education or an education degree
• 1-3 years of professional experience (or recent graduate with relevant internship experience)
• Passionate about solving education problems with technology
• Excited to explore different areas of tech (product, data analysis, quality assurance)
• Preferred: Startup or fast-paced environment experience

Application Notice:

Notice: Priority Deadline and Review Start Date

Please note that applications for this position will be accepted until 11/2/25 - applications received after this date will be reviewed on an intermittent basis. While we encourage early submissions, all applications received by the priority deadline will receive equal consideration. Thank you for your interest, and we look forward to reviewing your application.

Why Join Us?

• Work on cutting-edge AI technology that directly impacts educators and students.
• Join a mission-driven team passionate about making education more efficient and equitable.
• Flexibility of working from home, while fostering a unique culture built on relationships, trust, communication, and collaboration with our team - no matter where they live.
• Unlimited time off to empower our employees to manage their work-life balance. We work hard for our teachers and users, and encourage our employees to rest and take the time they need.
• Choice of employer-paid health insurance plans so that you can take care of yourself and your family. Dental and vision are also offered at very low premiums.
• Every employee is offered generous stock options, vested over 4 years.
• Plus a 401k match & monthly wellness stipend

Our Values:

• Educators are Magic: Educators are the most important ingredient in the educational process - they are the magic, not the AI. Trust them, empower them, and put them at the center of leading change in service of students and families.
• Joy and Magic: Bring joy and magic into every learning experience - push the boundaries of what’s possible with AI.
• Community: Foster community that supports one another during a time of rapid technological change. Listen to them and serve their needs.
• Innovation: The education system is outdated and in need of innovation and change - AI is an opportunity to bring equity, access, and serve the individual needs of students better than we ever have before.
• Responsibility: Put responsibility and safety at the forefront of the technological change that AI is bringing to education.
• Diversity: Diversity of thought, perspectives, and backgrounds helps us serve the wide audience of educators and students around the world.
• Excellence: Educators and students deserve the best - and we strive for the highest quality in everything we do.

Compensation Range: $80K - $130K