Data Integrity & Bias Detection: Verify training data integrity, perform bias detection, and conduct prompt-response validations for AI and GenAI model training.

AI Inter-Rater Reliability: Review and verify AI-generated outputs for consistency and accuracy. Collaborate with AI/ML engineers to enhance automated evaluation systems and pipelines.

Documentation & Reporting: Document test results, identify defects, collaborate to resolve issues, and prepare comprehensive reports with actionable insights for continuous improvement.

Compliance & Best Practices: Ensure adherence to industry standards, regulatory requirements, and Acentra Health standards around ethical and safe AI usage.

Continuous Improvement: Stay current with emerging AI testing methodologies and tools; contribute to evolving QA processes and strategies. Read, understand, and adhere to all corporate policies including policies related to HIPAA and its Privacy and Security Rules. The list of accountabilities is not intended to be all-inclusive and may be expanded to include other education- and experience-related duties that management may deem necessary from time to time.

Qualifications

Required Qualifications/Experience

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; equivalent work experience will be considered.
• 3–5 years of quality assurance experience, with a strong background in traditional QA and UAT methodologies.
• Proven experience in testing AI, ML, and Generative AI outputs, including Functional, UI, and API testing.
• Proficiency in RESTful APIs and testing tools such as Postman, specifically for testing GenAI model APIs like OpenAI and Hugging Face.
• Experience verifying training data integrity, bias detection, and prompt-response validation for AI models.
• Familiarity with data annotation tools, quality control processes, and GenAI-specific annotation for RAG and LLM fine-tuning.
• Demonstrated experience in testing complex software application.
• Proven experience designing, documenting, and executing comprehensive test plans for AI-driven products.
• Strong analytical thinking, attention to detail, excellent communication skills, and the ability to thrive in a fast-paced, cross-functional team environment.
• Strong experience with QA tools and frameworks like PyTest, Selenium, or JUnit, including Gen AI evaluation frameworks like RAGAS.
• Understanding of AI, ML, and GenAI concepts, algorithms, and output validation techniques.
• Familiarity with data annotation tools and quality control processes for GenAI-specific tasks.
• Solid knowledge of Python, with experience in Gen AI libraries like LangChain and LlamaIndex
• Experience with GenAI model evaluation metrics such as perplexity and BLEU.
• Solid knowledge of software development life cycle (SDLC) and agile methodologies.
• Excellent analytical and problem-solving skills with keen attention to detail.
• Strong communication and collaboration skills, with the ability to explain complex technical issues to diverse audiences.
• Ability to work effectively in a fast-paced, cross-functional team environment.

Preferred Qualifications/Experience

• Experience in healthcare technology or data analytics environments.
• Familiarity with regulatory and ethical standards related to AI and healthcare solutions.
• Why us?
• We are a team of experienced and caring leaders, clinicians, pioneering technologists, and industry professionals who come together to redefine expectations for the healthcare industry. State and federal healthcare agencies, providers, and employers turn to us as their vital partner to ensure better healthcare and improve health outcomes.
• We do this through our people.
• You will have meaningful work that genuinely improves people's lives across the country. We are a company that cares about our employees, and we give you the tools and encouragement you need to achieve the finest work of your career.
• Benefits
• Benefits are a key component of your rewards package. Our benefits are designed to provide you with additional protection, security, and support for both your career and your life away from work. Our benefits include comprehensive health plans, paid time off, retirement savings, corporate wellness, educational assistance, corporate discounts, and more.
• Thank You!
• We know your time is valuable and we thank you for applying for this position. Due to the high volume of applicants, only those who are chosen to advance in our interview process will be contacted. We sincerely appreciate your interest in Acentra Health and invite you to apply to future openings that may be of interest. Best of luck in your search!
• ~ The Acentra Health Talent Acquisition Team

Visit us at https: //careers.acentra.com/jobs EEO AA M/F/Vet/Disability Acentra Health is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran or any other status protected by applicable Federal, State or Local law. Compensation The pay for this position is listed below. “Based on our compensation philosophy, an applicant’s position placement in the pay range will depend on various considerations, such as years of applicable experience and skill level.”

Pay Range

USD $115,000.00 - USD $144,250.00 /Yr.