🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Cookies & analytics consent
We serve candidates globally, so we only activate Google Tag Manager and other analytics after you opt in. This keeps us aligned with GDPR/UK DPA, ePrivacy, LGPD, and similar rules. Essential features still run without analytics cookies.
Read how we use data in our Privacy Policy and Terms of Service.
🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Caseware • Perímetro Urbano Medellín, Antioquia, Colombia
Salary: 70%+ of critical pa
Role & seniority: AI Test Architect (senior level) at Caseware; fully remote, Colombia-based; reports to Jai Joshi.
Cloud/infra: AWS (serverless, microservices), IaC (Terraform/CloudFormation)
CI/CD & automation: GitHub CI/CD; Playwright/Cypress; AI-generated tests; self-healing automation
AI/LLM tooling & evaluation: LangChain/LangSmith/LangGraph, LangFuse, LangSmith, DeepEval, RAGAS, Arize Phoenix
Testing & evaluation: LLM evaluation tools, red-teaming concepts; tool-calling and multi-agent workflows
Data & governance: synthetic data generation, data masking, governance for ethical AI testing
Design and implement the Quality Intelligence platform using generative AI for defect prediction, test generation, self-healing automation, and SDLC integration.
Develop LLM/agent evaluation frameworks with benchmarks, red-teaming, adversarial testing, and observability; establish metrics (faithfulness, safety, bias) and governance.
Architect AI-enabled testing in CI/CD, build self-healing test frameworks, secures data/privacy, and drive cross-functional adoption of AI quality practices.
8+ years in Quality Engineering/Test Architecture for cloud-native SaaS; 2+ years in AI/ML/LLM testing
AWS (serverless/microservices) and Terraform/CloudFormation; GitHub CI/CD
Proficiency in JavaScript/TypeScript and/or Python
Experience designing/testing LLM-based apps and frameworks
Caseware is one of Canada's original Fintech companies, having led the global audit and accounting software industry for over 30 years, with more than 500,000 users across 130 countries and available in 16 different languages. While you might not have heard of us (yet) over 36,000 accounting and audit professionals list Caseware as a skill on their LinkedIn profiles!
Why This Role Matters As a leader in cloud-native SaaS, we are accelerating our shift to an AI-first future—embedding generative AI and autonomous agents across our platform to deliver smarter, faster user experiences. We are on the lookout for a visionary AI Test Architect to build the next-generation "Quality Intelligence" platform: one that leverages generative AI for automated test creation, self-healing execution, predictive defect analytics, and rigorous validation of our AI features built inhouse for our global audience.
As our foundational AI Test Architect, you'll design scalable, ethical frameworks that ensure reliability, safety, and compliance while accelerating release velocity (targeting 30-50% faster cycles through AI-augmented testing). Your work will reduce risk in production AI agents, minimize hallucinations/bias/security exposures, and empower the entire engineering organization to adopt AI-augmented quality practices that supplement traditional mature frameworks we have. This high-impact role sits at the intersection of Platform Engineering, AI, and Quality—shaping how we build trustworthy intelligence at scale.
📍 Location: This is a fully remote position located in Colombia.
What You’ll Be Doing
Evangelize and mentor: Upskill traditional QA engineers into AI-augmented testers through workshops, playbooks, and communities of practice. Drive adoption of AI quality best practices organization-wide, including metrics dashboards for DORA + AI-specific indicators (e.g., hallucination rate, red team success rate, self-healing coverage).
Challenges You'll Architect Solutions For Building reliable evaluation for non-deterministic, agentic AI in a fast-moving SaaS landscape. Scaling self-healing and generative test automation without introducing new flakiness or security debt. Balancing innovation speed with rigorous red teaming and ethical safeguards for customer-facing AI.
Success in the First 6-12 Months Launch the "Quality Intelligence" platform foundation with AI-augmented pipelines covering > 70%+ of critical paths. Establish red teaming/red-teaming-as-code processes that reduce high-severity AI risks by > 40%+. Upskill > 50%+ of QA/engineering teams on AI testing fundamentals and deliver measurable velocity/safety gains.
Accuracy Baseline: Establish a baseline 90%+ Faithfulness score for all RAG-powered features.
What You Will Bring 8+ years in Quality Engineering/Test Architecture within cloud-native SaaS environments, with 2+ years focused on AI/ML/LLM testing and validation. Deep expertise in AWS (serverless, microservices, IaC with Terraform/CloudFormation) and GitHub CI/CD ecosystems. Proficiency architecting LLM-based applications and testing frameworks (LangChain/LangGraph/LangSmith strongly preferred; equivalents acceptable). Mastery of modern automation (Playwright, Cypress) with hands-on experience integrating self-healing AI plugins or generative test tools. Strong programming skills in JavaScript/TypeScript and/or Python; solid understanding of foundational AI concepts (transformers, embeddings, RAG, evaluation trade-offs). Experience with LLM evaluation tools like Bedrock Evaluations, Prompt Management, Guardrails, DeepEval, RAGAS, Arize Phoenix, Langfuse. Experience with Red teaming frameworks/tools (Cobalt Strike, Sliver, Nmap) and knowledge of adversarial testing methodologies is a bonus.
Proven leadership: Mentoring teams, defining standards, and driving cross-functional change in ambiguous, high-growth settings. Bachelor's/Master's in Computer Science, AI/ML, or equivalent; relevant certifications a strong plus. Strong English language communication and collaboration skills
Perks & Benefits Contrato a termino Indefinido with all the legal benefits Prepaid Medicine Life insurance and funeral assistance Internet allowance Home office stipend Competitive compensation — above the market average 100% remote work environment and an excellent work-life balance Opportunity to work for a growing global SaaS leader company A culture that promotes independence, innovation, trust, and accountability Open space to be creative, innovative, and strategize for the future Mentorship by a highly experienced professional Budget for training, we want you to grow 5 Personal Time Off days per year Sick Leave Top up to total 100% of salary paid by the employer from Day 3 to 90. Recognition Award, additional paid time off in recognition of the corresponding year of service Upgrade vacation starting at 5 years of service
\n
▪️Innovation is at our core. We work with cutting-edge technology in accounting and financial reporting, constantly pushing the boundaries to create impactful software solutions. ▪️We are committed to a collaborative culture, where your ideas are valued, and knowledge sharing is encouraged within a supportive, inclusive team. ▪️Work-life balance is important to us. We offer flexible work options, remote opportunities, and generous time-off policies to ensure a healthy work-life balance. ▪️We offer competitive compensation, including a competitive salary and comprehensive benefits such as health insurance and retirement plans. ▪️We are driven by impactful work. Your contributions directly affect how our clients manage financial processes and drive their success. ▪️Recognition and rewards matter to us. We celebrate hard work through recognition programs, performance bonuses, and opportunities for career growth. ▪️We embrace global opportunities. Work on international projects and collaborate with a diverse, global team.
With a recent strategic investment from Hg Capital in 2020, Caseware is now in its next major growth phase as we double down on the people and products that have made Caseware so successful to date.
One of Caseware's core values is Many Voices, One Team and with that in mind, we're dedicated to building teams as diverse as our customers in an equitable and inclusive way. We welcome and encourage candidates of all backgrounds to apply. Should you require accommodations or have any questions at any point during the application or interview process, please e-mail our People Operations team at talent@caseware.com.