Senior Automation QA Engineer (Big Data Platform)
Grid Dynamics • Istanbul, Istanbul, Turkey
Role & seniority: Automation QA Engineer (senior/seasoned)
Stack/tools: Big Data stack (Apache Spark, HDFS, Flink; strong preference for Apache Iceberg); Python (automation/tests); SQL; cloud (AWS or equivalent); data analysis tools (Jupyter Notebooks, Apache Superset)
Top 3 responsibilities
-
Migrate Enterprise Data Platform from HDFS/Hive to Cloud Object Storage and validate migration outcomes
-
Develop and validate Spark-based ETL/ELT jobs (Scala/Python) for data movement to cloud storage
-
Create and maintain automated tests and integration scripts; perform functional testing and data quality checks; collaborate with Data Engineers to ensure data integrity
Must-have skills
-
Proven Automation QA experience on large-scale Big Data projects
-
Functional testing, code validation, end-to-end data verification
-
Python proficiency for automation/integration testing; strong SQL for data analysis
-
Experience with Spark (and ideally Flink), HDFS, and data quality verification in pipelines
-
Cloud familiarity (AWS or equivalent); excellent communication in distributed teams
Nice-to-haves
-
Apache Iceberg experience; experience validating data transformations with analytics tools (e.g., Jupyter, Superset)
-
Additional data governance, data quality frameworks, or CI/CD for data pipelines
Location & work type
-
Location: not specified in posting
-
Work type: not specified; offered as part of a professional role with flexible scheduling options
Full Description
We are seeking a seasoned Automation QA Engineer with a strong focus on Big Data to play a pivotal role in modernizing a massive Enterprise Data Platform. You will be instrumental in migrating an existing on-premises HDFS/Hive ecosystem to a modern Cloud Object Storage solution.
Responsibilities
Migration Support: Actively participate in the migration of the Enterprise Data Platform from HDFS/Hive to Cloud Storage.
Pipeline Development & Validation: Implement and validate Spark-based ETL/ELT jobs (Scala/Python) to facilitate data movement to cloud storage.
Automation: Develop and maintain robust automated tests and integration scripts, primarily utilizing Python.
Data Integrity: Conduct functional testing and code validation to ensure the quality and stability of data pipelines.
Verification: Test migration outcomes to confirm data accuracy, completeness, and reliability; perform critical data quality checks across systems.
Analysis: Use SQL and analytics tools (e.g., Jupyter Notebooks, Apache Superset) to analyze and validate processed or migrated data and interpret performance metrics.
Collaboration: Partner closely with Data Engineers to troubleshoot issues, validate fixes, and maintain exceptionally high data quality standards.
Requirements
Proven experience as an Automation QA Engineer, ideally within Big Data or large-scale data platform projects. Strong background in functional testing, code validation, and end-to-end data verification.
Big Data Tech Stack: Hands-on experience with Apache Spark, Flink, HDFS, and a strong preference for experience with Apache Iceberg.
Cloud Proficiency: Familiarity with AWS or equivalent major cloud environments.
Coding: Proficiency in Python for writing and maintaining robust integration and automation tests. Solid understanding of data analytics—capable of analyzing results and validating transformations using SQL. Experience in validating the quality and consistency of data and tools used within pipelines. Excellent communication and collaboration skills within distributed team settings.
We offer
Opportunity to work on bleeding-edge projects Work with a highly motivated and dedicated team Competitive salary Flexible schedule Professional development opportunities
About Us
Grid Dynamics (NASDAQ: GDYN) is a leading provider of technology consulting, platform and product engineering, AI, and advanced analytics services. Fusing technical vision with business acumen, we solve the most pressing technical challenges and enable positive business outcomes for enterprise companies undergoing business transformation. A key differentiator for Grid Dynamics is our 8 years of experience and leadership in enterprise AI, supported by profound expertise and ongoing investment in data, analytics, cloud & DevOps, application modernization and customer experience. Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India. Show more Show less
