Data Engineering_ Senior Test Engineer
- Jet2 Travel
- 4 - 8 years
- Pune
- 1 month ago
- Email to a friend
- Report this job
Job Description
Job Title Senior Test Engineer (SDET) Data Engineering
Job Type Permanent
Relevant Experience 5 to 8 years of experience in the field of QA Testing, Datawarehouse, Data Engineering, Big Data, Cloud & Analytics, Data Visualization
Reports into Team Lead - Data Engineering
Job Overview:
We are looking for a Senior Software Development Engineer in Test (SDET) to join our Data Engineering team. SDET has a dual role: They are both Data Engineers and quality assurance engineers. They write code to test software, focusing on creating automated tests, frameworks, and tools that help ensure the quality of data pipelines and data itself. The SDET works closely with Scrum teams and stakeholders to ensure the quality and reliability of complex data pipelines, ETL processes, and large-scale data systems. The role requires automating testing at various stages of the development lifecycle, from unit testing frameworks to system, integration, UAT and monitoring production data.
Roles and Responsibilities
Key Responsibilities:
- Collaborating with Scrum Teams:
- Actively participate in daily Scrum meetings, planning and refinement sessions to develop overall test strategy for Epic/Project
- Work closely with Product Owners, Business Analysts and Data Engineers to ensure that quality is baked into the design and development of data pipelines by identifying functional and non-functional test cases
- Engage with cross-functional teams like release/deployment, Infrastructure to ensure testing is integrated into the entire data engineering delivery process.
- Test Strategy and Framework Development:
- Design, develop, and implement unit testing frameworks for data pipelines, ETL processes, Data Visualization dashboards and distributed data systems.
- Develop automated testing frameworks for system, integration, and UAT.
- Define test strategies and best practices for ensuring accuracy, reliability, scalability, and performance of the data engineering solutions.
- Automation of Testing Processes:
- Automate tests for data validation, schema verification, data transformation, and data quality checks across different stages of the pipeline.
- Write scripts and programs in languages such as Python, Java, or Scala to automate test execution.
- Use advanced features in DBT to write generic and custom test cases and auto-execution.
- Implement automation test framework for testing and validating Data Visualization dashboards in Tableau.
- Ensure testing is integral part of continuous integration (CI) and continuous delivery (CD) practices to enable automation during the delivery lifecycle.
- Integration and System Testing:
- Ensure the quality of data ingestion, transformation, and storage systems by automating integration tests across multiple data sources and sinks.
- Perform system testing to validate the end-to-end data flow from data ingestion to final reporting and analytics systems.
- Collaborate with DevOps teams to ensure automated testing is integrated into the deployment pipelines.
- Performance and Load Testing:
- Design and implement performance tests to evaluate data pipeline throughput, latency, and scalability under various data volumes and conditions.
- Identify and resolve bottlenecks in data processing pipelines, working closely with Data Engineers to optimize performance.
- Data Visualization Testing:
- Test data accuracy & consistency in dashboards
- Visualization integrity & design testing
- Performance & load testing using Tableau
- Test Data Management:
- Create and manage test data sets for use in automation scripts, ensuring data variety and consistency for different testing scenarios.
- Develop strategies for handling large-scale test data, ensuring its accuracy and integrity during testing.
- Defect Identification and Tracking:
- Identify, document, and track defects found during testing.
- Work closely with development teams to ensure defects are resolved in a timely manner and that testing is adapted accordingly.
- Continuous Improvement and Innovation:
- Continuously evolve testing strategies and frameworks to keep up with new technologies, tools, and best practices in data engineering and quality assurance.
- Promote a culture of quality, innovation, and automation within the team.
Required Skills & Qualifications:
- Education: Bachelors degree in computer science, Data Engineering, or a related field. A Masters degree is a plus.
- Experience:
- 5+ years of experience in software testing, quality assurance, or data engineering roles, with a focus on automation.
- Experience in automating unit tests, system tests, and integration tests in data engineering environments.
- Hands-on experience with data pipelines, ETL frameworks, and data processing systems (e.g., Hadoop, Spark, Kafka, Airflow, etc.).
- Experienced in testing Data Visualization dashboards in Tableau preferably through automation.
- Technical Skills:
- Proficiency in programming languages: Python (particularly PyTest Or Nose2 Or Behave), Java(junit), Scala, or similar for developing automation scripts and frameworks.
- Proficiency with automation testing frameworks: Cucumber, Gherkin, TestNG, or similar for data testing
- Strong knowledge of SQL and experience in testing databases and data warehouses with DBT (e.g., Snowflake, Redshift, BigQuery).
- Strong Knowledge of workload automation platforms like Apache Airflow and DBT ( Data Build Tool)
- Familiarity with CI/CD tools (e.g., Jenkins, Azure CICD ) and experience integrating automated tests into pipelines.
- Experience with cloud platforms (AWS, GCP, Azure) for testing and deploying data solutions.
- Knowledge of performance testing and load testing tools such as Apache JMeter or Gatling.
- Familiarity with a modern Data Visualization tool such as Tableau.
- Data Engineering Concepts:
- Deep understanding of data warehousing, data modelling, ETL/ELT processes, and data quality assurance.
- Knowledge of big data technologies like Apache Spark, Hadoop, Kafka, and distributed data systems.
- Understanding of TDD and BDD for Data Engineering
- Agile Methodology:
- Experience working within Scrum teams and contributing to sprint planning, backlog refinement, and sprint reviews.
- Strong understanding of the Agile testing lifecycle and how to incorporate automated testing into the Agile development process.
Preferred Qualifications:
- Experience with test data management tools and strategies.
- Familiarity with data validation and data anomaly detection frameworks like Great Expectations or Deequ.
- Experience working in a DevOps environment with an emphasis on automating deployments and testing.
- Knowledge of security testing related to data privacy and compliance, particularly in sensitive data environments.
Soft Skills:
- Strong mindset of software testing and data quality. Focus on not just detecting defects but also on prevention
- Strong analytical and problem-solving skills, with an emphasis on attention to detail.
- Excellent communication and collaboration skills, especially within cross-functional teams.
- Ability to handle multiple tasks and projects in a fast-paced environment.
Job Classification
Industry: AviationFunctional Area / Department: Engineering - Software & QA
Role Category: Quality Assurance and Testing
Role: Data / Big Data Testing Engineer
Employement Type: Full time
Contact Details:
Company: Jet2 TravelLocation(s): Pune
Keyskills: snowflake python data software testing analytical data testing automation testing agile testing data warehousing cloud testing automation test framework sql data quality data modeling quality assurance collaboration etl programming communication skills
Similar positions
Jet2 Travel
Jet2 Travel Technologies Private Limited (J2TT) provides Information Technology (IT) services to Jet2.com and Jet2Holidays. J2TT works alongside the existing UK-based IT teams.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in