Full Description

Job Description: Phreesia is seeking a collaborative and proactive SDET 4 to help drive quality, reliability and efficiency across our AI-enabled VoiceAI platform, with a strong emphasis on testing AI agent behavior and leveraging AI in daily quality engineering workflows. In this role, using AI enabled automation you will play a key leadership role in ensuring the correctness, performance, and scalability of Voice AI agents and LLM-powered workflows that support patient–provider communication. You will design, build, and evolve automated functional, regression, and performance testing strategies tailored to AI-driven systems, while actively using AI tools to accelerate test design, test coverage analysis, and defect investigation. As an SDET 4, you will operate in a modern CI/CD environment using cloud services, where quality engineering must move as fast as product development. You will partner closely with senior SDETs, software engineers, product managers, and operations teams to embed quality into every stage of the development lifecycle—ensuring AI-driven features are reliable, observable, and safe to operate in production. If you’re excited about shaping testing strategies for AI agents in a real-world healthcare environment and about using AI as a force multiplier in your own work while holding systems to the highest quality bar, this role offers a unique opportunity to make a meaningful impact. Job Responsibilities Design, build, and maintain comprehensive automated test suites, iterating continuously within a CI/CD environment to ensure fast, reliable feedback Define and execute testing strategies for AI-enabled Voice AI agents and LLM-driven workflows, validating behavior, accuracy, performance, and reliability Conduct end-to-end testing across front-end, back-end, API, and performance layers to ensure system robustness in a distributed, message-based architecture Leverage AI-assisted tools and techniques to support test creation, test data generation, failure analysis, and exploratory testing where appropriate Perform root cause analysis for test and production failures, collaborating closely with engineers and SDETs to drive timely resolution Integrate automated testing into CI/CD pipelines and troubleshoot pipeline failures related to test execution and environment issues Assist in troubleshooting and resolving production escalations, contributing to system stability and long-term product quality Participate in code and merge request reviews for both application and test code, providing actionable feedback to uphold quality standards Monitor application performance, reliability, and operational health using AWS tools such as CloudWatch, X-Ray, and CloudTrail, identifying and addressing issues proactively Support products and services over their full lifecycle, from initial development through production and ongoing maintenance Test and validate systems built on cloud-native and serverless technologies, including AWS Lambda, SQS, SNS, DynamoDB, OpenSearch, as well as Azure and GCP services Collaborate with Product Managers, developers, and other SDETs to design effective test strategies and scenarios within an agile development process Test microservices and APIs in highly distributed systems, ensuring correctness, resilience, and observability Work with database architecture and operations, including DynamoDB, SQL-based systems, and NoSQL stores such as MongoDB Develop and maintain quality and test automation frameworks for .NET / .NET Core applications Continuously iterate on and improve test automation processes, frameworks, and tooling to increase coverage, reliability, and efficiency Proactively identify gaps in test coverage, quality risks, or tooling, and propose solutions to address them Create, maintain, and execute test plans and scenarios; curate high-quality test data; and monitor results to ensure comprehensive validation Mentor junior SDETs and QA engineers, sharing best practices in testing, automation, and effective AI-assisted quality engineering Qualifications Education Bachelor's Degree required, preferably in Computer Science Experience, Knowledge & Skills 5+ years of experience in Quality Assurance or as a Software Development Engineer in Test (SDET), testing complex, distributed systems 3+ years of professional programming experience using C# and TypeScript, including writing and maintaining automated API and integration test suites Hands-on experience testing or validating Voice AI agents and LLM-enabled workflows in production or pre-production environments Demonstrated experience leveraging AI-assisted tools to support testing activities such as test design, data generation, exploratory testing, and failure analysis Experience with infrastructure as code, containerization, and CI/CD pipelines (Docker, GitLab CI/CD preferred) Strong experience in UI automation using frameworks such as Playwright, Cypress, or Selenium Cloud experience with AWS, GCP, or Azure, with AWS strongly preferred Solid understanding of testing patterns, automation frameworks, and quality strategies across functional, integration, and performance testing Familiarity with modern web technologies, with a willingness and curiosity to learn new tools and frameworks as systems evolve Experience building and testing .NET / .NET Core–based applications in a cloud-native environment Technology Programming Languages: C#, TypeScript, .NET / .NET Core, Python — used for building test automation, API validation, and testing AI agent behaviors and LLM-driven workflows Testing Frameworks & Tools: Playwright, Selenium, xUnit, NUnit — for end-to-end, integration, and UI testing across traditional systems and AI-enabled Voice AI experiences CI/CD & Automation: GitLab CI/CD Pipelines — for integrating automated tests, AI-related validation checks, and quality gates into continuous delivery workflows Databases & Data Stores: MySQL, DynamoDB — for validating data integrity, test data management, and AI workflow persistence Cloud & Observability (AWS): Lambda, CloudWatch, ECS, EC2, DynamoDB, SQS — for testing, monitoring, and troubleshooting distributed, event-driven systems that support AI agents Total Cash Compensation for Canada-based employees ranges from $120,000-$140,000, inclusive of base salary and variable incentive, and is dependent on qualifications. In addition, Phreesia offers a a highly competitive and comprehensive Total Rewards package Disclosure: This posting is to fill an existing vacancy. Who We Are: At Phreesia, we’re looking for smart and passionate people to help drive our mission of creating a better, more engaging healthcare experience. We’re committed to helping healthcare organizations succeed in an ever-evolving landscape by transforming the way healthcare is delivered. Our SaaS platform digitizes appointment check-in and offers tools to engage patients, improve efficiency, optimize staffing, and enhance clinical care. Phreesia cares about our employees by providing a diverse and dynamic work environment. We’re a five-time winner of Modern Healthcare Magazine’s Best Places to Work in Healthcare award and we’ve been recognized on the Bloomberg Gender Equality Index. We are dedicated to continuously improving our employee experience by launching new programs and initiatives. If you thrive in a culture of recognition, value inclusivity, professional development, and growth opportunities, Phreesia could be a great fit! Top-rated Employee Benefits: Remote First: 100% Remote work + home office expense reimbursements+ monthly reimbursement for cell phone, internet and wellness. Top of market rewards: Competitive compensation Take time when you need time: Flexible PTO + company holidays Top class healthcare benefits: Variety of healthcare benefits for you and your family (and your pets!) starting day one Care about your families: Generous top-up for parental leave benefits Support personal development: Continuing education and professional certification reimbursement Connecting in person: Various offsite events and activities for team to connect and meet in person, to support team building and engagement. Giveback to community: Local in-person volunteer events, and give back programs to our communities. Recognition and perks: We have a company wide recognition tool (Phireworks) to celebrate milestones, recognize achievements and strengthen your bond with your teams. You can accumulate points and redeem them for a wide catalogue of items! Diversity and inclusive environment: At Phreesia, all employees are encouraged to bring their authentic self to work, feel supported and perform at their best. We have a variety of Employee Resources Groups (ERGs) which bring together individuals from a wide range of backgrounds, experiences and perspectives, and seek to foster a sense of shared community and empowerment for employees who share a common social identity, such as gender, race, ethnicity, and sexual orientation. Opportunity to join an Employee Resource Group. Learn more here: Disclosure: Phreesia uses certain automated tools, including artificial intelligence, to support the assessment of applicants for this position. We strive to provide a diverse and inclusive environment and are an equal opportunity employer. At Phreesia, we’re transforming the patient experience and giving healthcare organizations the capacity to do more—and we need smart, innovative problem-solvers to help us get there. Phreesians have a wide range of backgrounds—you don’t need to be a healthcare expert or a technology geek to succeed on our team. We’re looking for talented, diverse individuals who want to make a difference in healthcare.