AI Test Engineer

Description

Data Science problems are everywhere, but the talent is not. At ObviouslyAI, our vision is to turn every company into an AI company. We do this by providing businesses with access to world-class, on-demand data science talent that helps them solve real business problems. On the back end, we empower data scientists with a set of internal groundbreaking tools to help them deliver results in minutes, not months.

We’re a small, scrappy group of people with a strong bent toward failing fast, a bias for action, and attention to detail. We’re focused on doing the best work of our lives and believe in having a healthy separation of work and life. We keep working hours flexible and are building a team with business teams located in San Francisco, CA and engineering teams located in Bangalore, India.

Obviously AI is backed by some of the top venture capital firms in the US, and you’ll be on the ground floor of a fast-growing company with a big mission.

About You

We’re looking for a creative and curious Backend & AI Agent Testing Engineer to join our team. You’ll work hands-on with our AI agents and backend services, writing code, debugging, scripting tests, and evaluating LLM prompts to ensure our systems behave reliably — even in unpredictable scenarios.

This is not a traditional backend or QA role. You’ll quickly learn new tools, explore unfamiliar apps, and design tests from a user’s perspective to help tame and improve our cutting-edge AI systems.

Responsibilities

• Build and Maintain
• Write, debug, and maintain backend code in Python or JavaScript, building test cases and backend scripts to support robust and experimental systems.
• Implement APIs and ensure authentication workflows work as expected, with exposure to integrations.
• AI Agent Testing & Prompt Evaluation
• Design and execute creative test strategies for AI agent behavior, particularly around LLM-based (GenAI) agents, ensuring systems behave reliably despite their unpredictable nature.
• Evaluate AI agent outputs and prompts, contributing to LLM evaluation and metrics using tools like Deepchecks.
• Scripting & Automation
• Write scripts and automation to test AI agents and backend workflows.
• Build lightweight automation frameworks and develop or extend test infrastructure; familiarity with Selenium or similar is a plus.
• Experimentation & Exploration
• Dive into new, unfamiliar apps and services quickly — learning them and building tests as if you were an end-user.
• Get into the “user’s shoes” to anticipate edge cases and potential failure modes.
• Tame the “beast” — creatively managing and testing AI systems that may behave inconsistently.
• Tools & Integrations
• Work hands-on with integrations (e.g., HubSpot, Salesforce — bonus points for experience here) and modern collaboration tools like Cursor or Windsurf.
• Test and validate backend workflows that connect names, emails, and other critical user data across systems.
• Collaboration & Continuous Learning
• Collaborate closely with cross-functional teams in a startup environment, embracing rapid experimentation and iteration.
• Continuously learn new frameworks, tools, and approaches without handholding, demonstrating a strong growth mindset.

Requirements

• Experience Level: ~0–3 years in backend development or testing, ideally in a startup or experimental role.

• Backend Engineering Basics: Able to write backend code, debug, and write test cases in Python or JavaScript.
• Testing Creativity: Not a traditional tester — you think creatively, experiment boldly, and approach testing like teaching a child.
• AI Agent Experience: Exposure to or experience testing and prompting AI agents, especially GenAI/LLM-based systems.
• Automation & Scripting: Comfortable writing backend scripts, automating tests, and creating testing frameworks.
• Integrations & APIs: Exposure to integrations (bonus if HubSpot, Salesforce), understanding of API interactions and authentication.
• Tools: Familiarity with tools like Cursor or Windsurf preferred.
• Knowledge of modern automation frameworks (e.g., Selenium).
• Experience in B2B product environments.
• LLM Evaluation & Metrics: Bonus if you’ve worked with LLM evaluation frameworks, metrics, or MCPs.
• Mindset:

• Curious and fast learner — you’re comfortable diving into completely unknown tools or apps and figuring them out.
• Super creative in designing tests beyond “clicking around” — you understand that testing AI systems means “taming the beast.”
• Willing to experiment, work on ambiguous problems, and wear multiple hats.

Benefits

• Work at the intersection of backend engineering, AI, and creative testing.

• A fast-moving, supportive startup culture that values experimentation and creativity.
• Opportunity to work with cutting-edge AI systems, tools, and frameworks.
• Learn and grow rapidly alongside a talented and collaborative team.