🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Cookies & analytics consent
We serve candidates globally, so we only activate Google Tag Manager and other analytics after you opt in. This keeps us aligned with GDPR/UK DPA, ePrivacy, LGPD, and similar rules. Essential features still run without analytics cookies.
Read how we use data in our Privacy Policy and Terms of Service.
🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Siemens • Noida, Uttar Pradesh, India
Role & seniority
Stack/tools
Python testing: Pytest; REST Assured; Selenium; API testing
CI/CD: Jenkins; GitHub Actions
Cloud/ML: AWS SageMaker, EKS, Lambda; Gen AI / AI-assisted automation tools
Testing concepts: ML/AI systems testing, LLM QA, AI-driven defect analysis
Others: AI-powered evaluation harnesses, self-healing automation, CI/CD-integrated QA
Top 3 responsibilities
Lead and execute testing of LLM, AI, and ML applications; ensure coverage and reliability
Develop and maintain AI-assisted test scripts; leverage Gen AI for faster generation and self-healing
Apply data-driven analysis and AI insights; perform automated testing, defect analysis, and robust reporting
Mentor associate engineers on AI-driven QA practices; collaborate cross-functionally to shift-left testing
Investigate defects for root cause using AI-powered anomaly detection; maintain intelligent defect tracking
Must-have skills
Bachelor’s in CS or related field (or equivalent experience)
3–4 years in test automation/QA
Python testing frameworks (Pytest), Selenium; API testing
CI/CD experience (Jenkins, GitHub Actions)
AWS familiarity (SageMaker, EKS, Lambda); data structures, SQL, OOP
Experience or willingness to learn AI-assisted automation / Gen AI QA
Strong communication and collaboration; fundamentals of ML/AI, ML lifecycle
Experience with AI model monitoring pipelines, prompt engin
Who We Are
Build a brighter future while learning and growing with Siemens company at the intersection of technology, community, and sustainability. Our global team of innovators is always looking to create meaningful solutions to some of the toughest challenges facing our world. Find out how far your passion can take you.
About The Job
Brightly Software continues to grow and needs amazing engineers ready to embrace the future of AI-powered Quality Assurance! This is an excellent fit for talented engineers who thrive in a fast-paced environment and are passionate about testing Generative AI and traditional ML applications using Generative AI and advanced automation to redefine software quality. You will work alongside top-notch engineers and product teams to design, implement, deliver, and support highly ambitious products and integrations, while championing modern QA practices.
The AI Test Engineer (Automation) supports application platforms in the Engineering department. This role is critical to overall program success, with aggressive timelines and direct impact on our customers. You will play a key role in implementing AI-assisted automation frameworks and driving intelligent testing strategies.
What You’ll Be Doing
Lead and execute thorough testing of LLM, AI, and ML applications, ensuring comprehensive coverage and reliability across diverse platforms and use cases.
Demonstrate strong inclination towards product quality and AI-first testing mindset.
Develop and maintain AI-assisted test scripts from functional and technical designs, leveraging Gen AI tools for faster script generation and self-healing capabilities.
Apply data-driven analysis and AI-powered insights for pragmatic decision-making aligned with business and technology needs.
Understand user requirements and implement intelligent test case generation for ML and AI systems using Gen AI platforms.
Perform automated testing, debug automation code, and provide comprehensive reports enriched with AI-driven defect analysis.
Work on improving test automation coverage using next-gen automation platforms that integrate AI co-pilots.
Collaborate with cross-functional teams to ensure product quality and advocate for AI-supported shift-left testing.
Investigate defects for root cause using AI-powered anomaly detection.
Execute test cases, log defects, and maintain test documentation with intelligent defect tracking systems.
Mentor associate engineers on AI-driven QA practices and automation best practices.
What You Need
Bachelor’s in Computer Science or related discipline; or equivalent work experience.
3–4 years of experience in test automation and QA.
Strong knowledge of Python testing frameworks such as Pytest, REST Assured, Selenium, API testing, CI/CD pipelines such as Jenkins and Github Actions.
Solid understanding of AWS services such as Sagemaker, EKS and lambda.
Solid understanding of data structures, algorithms, OOP, databases, SQL, and web programming.
Hands-on experience with AI-assisted automation tools or willingness to learn emerging Gen AI QA platforms.
Strong communication and collaboration skills.
Fundamentals of ML and AI systems such as the ML Lifecycle, model monitoring, testing and testing of LLM applications.
Experience validating AI model monitoring pipelines for drift, degradation, and anomaly signals.
Experience with prompt engineering and evaluation frameworks (e.g., DeepEVAL, OpenAI Evals).
Ability to design evaluation metrics for LLM‑generated outputs, including hallucination detection, grounding accuracy, factual consistency, and robustness testing.
Familiarity with ethical AI testing: bias detection, toxicity testing, safe‑prompt evaluations, and guardrail validation.
Hands‑on expertise building automated evaluation harnesses for GenAI outputs integrated within CI/CD pipelines.
Exposure to self‑healing automation frameworks enhanced by LLMs.
Knowledge of LLM failure patterns, adversarial prompt testing, and structured red‑teaming.
What Makes You Standout
Exposure to public cloud platforms (AWS, GCP).
Certifications like ISTQB, Python, or AI/ML fundamentals.
Experience with AI-powered defect prediction or self-healing automation frameworks.
The Brightly Culture
We’re guided by a vision of community that serves the ambitions and wellbeing of all people, and our professional communities are no exception. We model that ideal every day by being supportive, collaborative partners to one another, conscientiously making space for our colleagues to grow and thrive. Our passionate team is driven to create a future where smarter infrastructure protects the environments that shape and connect us all. That brighter future starts with us.