Sr. Data Engineer @ IBM India

Job Description

Introduction
At IBM, work is more than a job it s a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better, but to attempt things you ve never thought possible. Are you ready to lead in this new era of technology and solve some of the world s most challenging problems If so, lets talk.
Your Role and Responsibilities
We re looking for an experienced, motivated hands-on data engineer who brings ideas about handling largescale enterprise applications leveraging data platforms; As a Senior software engineer, you ll apply your deep expertise in designing, developing, delivering, and supporting a world class software and data platform. You will take full ownership of delivering high-impact big data platform that is robust, scalable and support production-grade applications and services for the supply chain space. You will leverage open source and cloud storage tools to build and develop reusable components and architecture that can enable the data science teams to provide best in class AI/ML and data analysis environment.
You will also help in providing technical direction and develop strategies for long-term platform growth. You need to be versatile, display leadership qualities and open minded to take on new problems that our customers face.
The day today responsibilities include:

• Analyzes and designs reusable components of the data platform and services required to support the data storage, data schema, data orchestration.
• Design, develop, troubleshoot, and scale the data pipelines required to support the various analytics and AI/ML workloads.
• Understand application produced artifacts, design the entire pipeline of schema definition, efficient storage and query of various entity objects.
• Translate complex technical and functional problems into detailed designs
• Partner and work with data scientists in the team in taking data science algorithms and integrating them efficiently for high scale production application.
• Provide senior level support and mentoring by evaluating product enhancements for feasibility studies and providing completion time estimates
• Develop high quality unit, tests functional tests and integration tests supporting the data extract, transform, load pipelines
• Ensure product quality by participating in design reviews, code reviews and working with the team for end-to-end validation of the entire product
• Design and develop various data validation strategies ensuring that robust , good quality data is provided to data science teams for model development and advanced analytics
• Define data governance, data auditing policy and strategies for compliance and security controls
• Write and maintain technical documentation for the various projects. Review product user documentation for technical accuracy and completeness

Required Technical and Professional Expertise

• 7-8 years of experience in developing enterprise applications using Java, Python, spark and related technologies with 2+ years a focus on DataEngineering, DataOps, MLOps
• Software development strategies for low latency, high throughput softwares
• Hands-on experience with common distributed processing tools and languages Python, Spark, Hive, Presto
• Deep understanding of data pipelines, data modeling strategies, schema management
• Experience with specialized data architectures like data lake, data mesh and optimizing data layouts for efficient processing.
• Hands on Experience with streaming platforms and frameworks like Kafka, spark-streaming
• Strong understanding of advanced algorithms used in design and development of enterprise grade software
• Familiarity with pipeline orchestrator tools like Argo, Kubeflow, Airflow or other open source
• Familiarity with platforms like Kubernetes and experience building on top of the native platforms
• Good written and verbal communication skills
• Ability to provide guidance to less experienced team members.

Preferred Technical and Professional Expertise

• Understanding of DevOps principles and experience building a and designing a DevOps first way of working
• Experience working with teams with focus on supply chain data

Employement Category:

Employement Type: Full time
Industry: IT Services & Consulting
Role Category: General / Other Software
Functional Area: Not Applicable
Role/Responsibilies: Sr. Data Engineer

+ View Contact

Keyskills:   sql java data warehousing informatica python big data open source low latency supply chain data science data analysis data modeling cloud storage data validation product quality data governance high throughput model development commercial models security controls