← All roles

Senior QA Analyst

AI & Apps · Remote, USA or Canada · Full time

Overview

We’re looking for a hands-on QA Analyst to help our team complete an AI-driven, character-based experience where users have live, contextual conversations in an entertainment setting.


Key Responsibilities

QA & Coverage

  • QA testing including structured testing, freeform conversations, and regression.

  • Ensure full coverage across all scenes and track progress against milestones.

  • Clearly separate technical bugs from AI response issues, and route them to the right teams.

  • Define and evolve quality standards for AI responses (accuracy, character voice, relevance, and repetition).

Active Testing

  • Run structured test sessions on Android devices using defined testing frameworks.

  • Evaluate AI responses, identify regressions, and document findings in a clear, actionable way.

  • Review transcripts and logs to spot patterns like hallucinations, fallbacks, or out-of-character behavior.

  • Work closely with a small group of developers to ensure their changes are submitted cleanly and don’t cause unwanted side effects.

  • Test builds, file bugs, and assess AI output against the knowledge base and prompt setup.

  • Support stress and edge-case testing, including rapid context switching and adversarial scenarios.

  • Test and identify issues with prototype hardware.

  • Test to ensure Publisher requirements are met before submission for certification

Qualifications

Key Experience and Qualities

  • Experience in a QA Analyst role, ideally in mobile, interactive media, or AI-driven products.

  • Hands-on experience testing AI/LLM systems, including evaluating response quality and consistency.

  • Strong mobile QA background, especially on Android (debugging tools, APK installs, etc.).

  • Clear, concise communicator who can report effectively to both technical and creative teams.

  • Highly organized, with the ability to manage multiple testing streams and collaborators.

  • Comfortable with tools like Jira, Google Workspace, and Slack.

Nice to Haves

  • Experience with automated AI evaluation or LLM-as-judge frameworks.

  • Familiarity with platforms like Vertex AI or Firebase.

  • Experience testing voice or conversational systems.

  • Exposure to XR/AR/MR or device-based testing.

  • A strong editorial instinct — able to judge tone, voice, and narrative quality, not just correctness.

  • Experience working with external QA vendors.

Why Join Us?

  • Work at the intersection of film, AI, and immersive tech.

  • Have real ownership over quality on the end product.

  • Work with a small but nimble team on cutting edge technology with some of the biggest clients in the tech and film space. 

If you know QA, understand AI, and like building things from the ground up, we’d love to talk.