
Test Automation Specialist
Tror - AI for everyone โข United States
**Role & seniority: ** Test Automation Specialist (senior; 9+ yrs testing, 6+ yrs automation framework design)
**Location & work type: ** Remote
**Stack/tools (primary): **
-
Playwright (TypeScript preferred)
-
TypeScript, Python, Java
-
AI/LLM & agents: AI agents, orchestration, prompt engineering, context management, tool calling, retrieval/grounding, LLM failure analysis, AI/LLM evaluation frameworks
-
CI/CD: Jenkins, Azure DevOps
-
Version control: Git, GitHub, Bitbucket
-
GitHub Copilot custom agents/instructions
-
Enterprise testing: SDLC/STLC, Agile
-
Experience with AI governance/security/compliance
**Top 3 responsibilities: **
-
Design/maintain Playwright automation frameworks and coding standards
-
Build UI/API/DB/E2E automated tests and integrate them into CI/CD
-
Develop AI agent/orchestration workflows and LLM evaluation/quality metrics
**Must-have skills: **
-
9+ yrs software testing; 6+ yrs framework design
-
Playwright expertise (TS preferred)
-
AI Agents/LLM-based tooling development + strong evaluation/metrics
-
CI/CD integration (Jenkins/Azure DevOps)
-
Git + framework architecture/shared tool development
-
Strong technical communication/leadership/mentoring
-
Prompt engineering + context management + retrieval/grounding + tool calling + LLM failure analysis
**Nice-to-haves: **
-
Acentra experience
-
EvoBrix experience
-
Preference
Full Description
Test Automation Specialist
Remote
Job Summary
We are seeking a Test Automation Specialist to design, build, and support AI-driven test automation solutions. This role combines deep Playwright automation expertise with AI Agent development, LLM evaluation, and automation framework architecture to support enterprise-wide QA modernization initiatives.
Key Responsibilities
Design and maintain Playwright automation frameworks. Create UI, API, Database, and End-to-End automated tests. Build AI agents and orchestration workflows for automation assets. Define coding standards and framework best practices. Integrate automation into CI/CD pipelines. Establish AI evaluation frameworks and quality metrics. Package and distribute reusable automation solutions. Provide technical leadership and mentoring. Ensure AI governance, security, and compliance standards.
Mandatory Skills
9+ years of Software Testing experience 6+ years of Test Automation Framework Design AI Agents / LLM-based Tools Development GitHub Copilot Custom Agents & Instructions Playwright (TypeScript Preferred) TypeScript, Python, and Java Prompt Engineering & LLM Concepts Context Management Tool Calling Retrieval/Grounding LLM Failure Analysis AI/LLM Evaluation Frameworks CI/CD (Jenkins, Azure DevOps) Git, GitHub, Bitbucket Framework Architecture & Shared Tool Development SDLC, STLC & Agile Methodologies Strong Communication & Technical Leadership Provider Testing Background
Acentra experience
EvoBrix experience
Resources that have all 3 are preferred, but 2 out of 3 could work as well.