🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Cookies & analytics consent
We serve candidates globally, so we only activate Google Tag Manager and other analytics after you opt in. This keeps us aligned with GDPR/UK DPA, ePrivacy, LGPD, and similar rules. Essential features still run without analytics cookies.
Read how we use data in our Privacy Policy and Terms of Service.
🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
ChatGPT Jobs • Seattle, Washington, United States
Role & seniority: Senior AI Quality Assurance Analyst (7+ years QA experience; focus on AI/ML & enterprise integrations)
Stack/tools: Selenium/Cypress; Postman/REST-assured; PyTest/JUnit; CI/CD (Azure DevOps, Jenkins); Git; API/UI test automation; monitoring/logging (Azure Monitor, Splunk); experience with Dynamics CRM, QicLink claims platform, Docuvantage
Design, write, and automate test cases for AI outputs and traditional features; establish pass/fail criteria and use statistical sampling when appropriate
Own regression testing for AI capabilities; continuously re-run critical scenarios as models/data change
Conduct adversarial/edge-case testing (stress tests, prompt injections); document findings and drive fixes with AI Product Owner and SMEs
7+ years in software QA/testing for enterprise or complex integrations
Proven test automation experience (UI and API) and regression suites in CI/CD
Familiarity with AI/ML system testing principles; ability to validate variable AI outputs and set acceptable accuracy thresholds
Strong analytical, documentation, and cross-team communication
Experience with testing tools (Selenium/Cypress, Postman/REST-assured), frameworks (PyTest/JUnit), and CI/CD integration; version control (Git)
Industry experience in insurance or healthcare
Performance testing and security testing tools for AI systems
Location & work type: Remote (ba
Job Description
AI Quality Assurance Analyst
Company: Allied Benefit Systems
Location: Chicago, IL
Remote
Type: Full-time
Benefits: Medical, Dental, Vision, Life, PTO
Posted: 15 hours ago
Job Description
POSITION SUMMARY
The AI Quality Assurance Analyst is the quality champion for Allied's emerging AI capabilities, ensuring that every model, integration, and automation works reliably and safely within our business processes. Working alongside developers, product owners, and business users, the Analyst will design tests that push the limits of our applications and help deliver trustworthy AI solutions.
Essential Functions
Test AI features integrated into core systems (e.g., Dynamics CRM, QicLink claims platform, Docuvantage) to ensure they function correctly in end-to-end workflows. Write and automate test cases for AI outputs and traditional software features; define clear pass/fail criteria (using statistical sampling when appropriate) to validate AI results.
Own regression testing for AI capabilities: continuously re-run critical test scenarios as models are updated or data changes, to catch issues early. Contribute to the overall test strategy and perform adversarial or edge-case testing (e.g., stress tests, prompt injections) to evaluate the robustness and security of AI components. Document and recommend solutions for issues found; coordinate with the AI Product Owner and SMEs to clarify requirements for fixes and improvements. Collaborate closely with developers and the Product Owner to report bugs, verify fixes, and improve requirements. Emphasize shared accountability for quality across the team.
EDUCATION
Bachelor's degree in related field or equivalent work experience required.
Experience And Skills
7+ years in software Quality Assurance/testing roles, with significant experience in enterprise systems or complex integrations required. Proven expertise in test automation (UI and API) and building regression test suites in CI/CD environments. Familiarity with AI/ML system testing principles. Comfortable validating outputs that may vary statistically and determining acceptable accuracy thresholds. Strong analytical skills to identify edge cases and pinpoint defects across interconnected systems (frontend, backend, data, model). Excellent documentation and communication skills for writing test plans, reporting issues, and guiding teams on quality best practices. Experience in industries like insurance or healthcare is a plus (provides domain context for Allied's processes). Test automation tools such as Selenium or Cypress for web interface testing, and Postman or REST-assured for API testing. Testing frameworks like PyTest or JUnit, integrated with CI/CD pipelines (Azure DevOps, Jenkins) for continuous testing. Version control (Git) for maintaining test scripts, and monitoring/logging tools (Azure Monitor, Splunk) to investigate issues. Performance testing or security testing tools for evaluating AI systems under load or threat scenarios.
POSITION COMPETENCIES
Job Knowledge Time Management Accountability Communication Initiative Customer Focus
Physical Demands
This is a standard desk role - long periods of sitting and working on a computer are required.
WORK ENVIRONMENT
Remote
Here at Allied, we believe that great talent can thrive from anywhere. Our remote friendly culture offers flexibility and the comfort of working from home, while also ensuring you are set up for success. To support a smooth and efficient remote work experience, the internet connection must be obtained through a cable broadband or fiber optic internet service provider with speeds of at least 100Mbps download/25Mbps upload. Reliable internet service is essential for staying connected and productive.
Compensation is not limited to base salary. Allied values our Total Rewards, and offers a competitive Benefit Package including, but not limited to, Medical, Dental, Vision, Life & Disability Insurance, Generous Paid Time Off, Tuition Reimbursement, EAP, and a Technology Stipend.
Protect Yourself from Hiring Scams
Important Notice About Our Hiring Process
All interviews are conducted via video. No job offer will ever be made without a video interview with Human Resources and/or the Hiring Manager.
If someone contacts you claiming to represent us and offers a position without a video interview, it is not legitimate. We never ask for payment or personal financial information during the hiring process.
For your security, please verify all job opportunities through our official careers page: Current Career Opportunities at Allied Benefit Systems
Your security matters to us-thank you for helping us maintain a fair and trustworthy process!