Role & seniority: Data quality automation engineer (1+ year experience), likely individual contributor.

Stack/tools: PySpark, Python, SQL, Hive; Hive/HDFS; CI/CD pipelines; experience with data quality frameworks (preferred: Deequ, Great Expectations); workflow tools (preferred: Airflow, Step Functions).

Top 3 responsibilities

Design, develop, and maintain automated data quality checks for petabyte-scale data pipelines.
Implement scalable data validation frameworks and integrate automated tests into CI/CD workflows; analyze results and resolve data quality issues.
Develop monitoring/alerting for data quality in production; document processes and mentor junior engineers; continuously improve frameworks.

Must-have skills

Bachelors or Masters in CS/Engineering or related field.
1+ year in test engineering or data engineering focused on data quality and automation.
Proficiency in PySpark and Python; strong SQL and Hive; hands-on with Hive/HDFS.
Understanding of data pipelines/ETL and experience with CI/CD for automated testing.
Strong analytical, problem-solving, and communication skills; ability to work independently and in a team.

Nice-to-haves

Experience with data quality frameworks (Deequ, Great Expectations).
Familiarity with Airflow or Step Functions; data cataloging, lineage, and metadata management.
Location & work type: Hybrid role; in-office days TBD by hiring manager.

Full Description

Company Description Visa is a world leader in payments and technology, with over 259 billion payments transactions flowing safely between consumers, merchants, financial institutions, and government entities in more than 200 countries and territories each year. Our mission is to connect the world through the most innovative, convenient, reliable, and secure payments network, enabling individuals, businesses, and economies to thrive while driven by a common purpose – to uplift everyone, everywhere by being the best way to pay and be paid. Make an impact with a purpose-driven industry leader. Join us today and experience Life at Visa. Job Description

Responsibilities

Design, develop, and maintain automated data quality checks for complex data pipelines handling petabyte scale datasets.
Implement scalable data validation frameworks using PySpark, PyTest , Python, SQL, and Hive, ensuring comprehensive test coverage.
Collaborate with Data Engineers and DevOps teams to integrate automated tests into CI CD workflows.
Analyze test results, identify data anomalies, and provide actionable insights to resolve data quality issues.
Develop monitoring and alerting solutions for data quality in production environments.
Document test processes, standards, and best practices mentor junior engineers on data quality automation.
Continuously improve test frameworks and processes to optimize performance, scalability, and reliability.
This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.
Qualifications

Basic Qualifications

Bachelors or Master's degree in Computer Science, Engineering, or related field.
1+ years of experience in test engineering or data engineering roles, with a focus on data quality and automation.
Bachelors or Master's degree in Computer Science, Engineering, or related field.
1+ years of experience in test engineering or data engineering roles, with a focus on data quality and automation.
Advanced proficiency in PySpark and Python for building scalable data processing and testing solutions.
Strong experience with SQL and Hive for querying, validating, and profiling data in large datasets.
Hands on exposure to Hive and HDFS.
Solid understanding of data pipeline architectures, ETL processes, and best practices for big data environments.
Experience implementing CI CD pipelines for automated data testing.
Strong analytical, problem solving, and communication skills.
Ability to work independently and in a collaborative, fast paced team environment.

SW Test Engineer