Data Engineer - AllSpark @ Michael Page

Job Description

• Work with Business & IT teams to gather, understand and define requirements and break down Demands into EPICS and User Stories, where needed.
• Lead the design to build end-to-end CI/CD data pipelines to get the relevant data and for the deliverables
• Reviewing design, code and other deliverables created by your team to guarantee high-quality results
• Define acceptance criteria, manage & perform testing of own pipelines to ensure expected quality

• Own the status of Change requirements throughout the lifecycle & adapt changes to the existing scripts, codes and pipelines.
• Own PoCs and deliver the results in reasonable time
• Ensuring appropriate use of MB IT project management tools, governance, processes, policies and methodologies
• Share required data for regular status meetings with all stakeholders, keeping the stakeholders needs and requirements continuously in view
• Bring value adds by contributing to continuous improvement and innovation, encourage best practices, challenge current practices, provide feedback to colleagues
• Presents user stories in demand concept meetings together with respective stakeholders
• Lead/support architectural tasks if and as needed
• Detect dependencies to other systems/demands
• Conduct regular status meetings with all stakeholders, keeping the stakeholders needs and requirements continuously in view
• Create / Propose Demand Management baselines as and when required.

• Total 10-12 Years of Experience with minimum 4 years of experience in data engineering & analytics.
• Bachelors Degree in computer science or equivalent
• Expertise in Azure and Azure Data Factory, Storage accounts and Notebooks
• Good experience with DWH, ETL/ELT process, hands-on expertise ETL tools, relational DB or MPP
• Expertise in Python, structured query language, PL/SQL, query tuning and performance tuning
• Good knowledge in DevOps, CI/CD including deploying a range of data engineering pipelines into production, and testing techniques
• Expertise in Big Data technologies such as Apache Spark, Hadoop ecosystem, Apache Kafka, NoSQL databases, etc.
• Good knowledge of data architecture patterns (Data lakehouse, delta lake, streaming, Lambda/Kappa architecture)
• Agile development experience
• Experienced in Powershell scripting for orchestration

Employement Category:

+ View Contact

Keyskills:   sql java data warehousing informatica python it project management big data etl tools user stories query tuning apache kafka apache spark computer science data engineering data architecture demand management project management continuous improvement powershell scripting statements of work sow