Job Description
Key Responsibilities:
- Architecture & Design
- Design scalable, high-performance, cloud-native data architectures using Azure Data Lake, Azure Synapse, and Databricks.
- Develop high-level and low-level architecture documents (HLD/LLD) for modern data platforms.
- Define data models using star and snowflake schemas, optimizing for analytics and query performance.
- Data Engineering & ETL
- Lead the development of ETL/ELT pipelines using Azure Data Factory, PySpark, Spark SQL, and Databricks.
- Manage ingestion of structured and semi-structured data from diverse sources to Azure-based data lakes and warehouses.
- Implement real-time data pipelines using Azure Event Hubs and Structured Streaming.
- Governance & Security
- Define and implement data governance frameworks including lineage, cataloging, access controls, and compliance (e.g., GDPR).
- Collaborate with MDM and governance teams using tools like Informatica AXON and EDC.
- Performance Tuning & Optimization
- Drive cost-efficient architecture design with partitioning, caching, indexing, and cluster optimization.
- Monitor and troubleshoot data pipelines using Azure Monitor, Log Analytics, and Databricks tools.
- Stakeholder Engagement
- Collaborate with data scientists, analysts, business stakeholders, and DevOps teams to deliver robust, scalable data platforms.
- Conduct design reviews and training sessions to support platform adoption and knowledge sharing.
Roles & Responsibilities
Key Skills & Technologies:
Cloud Platforms: Azure (ADF, ADLS, Azure SQL, Synapse, Databricks), AWS (S3, RDS, EC2)
Big Data: Spark, Delta Lake, PySpark, Hadoop
ETL Tools: Azure Data Factory, Informatica, IBM DataStage
Data Modeling: Star, Snowflake, SCD, Fact & Dimension Tables
Programming: Python, PySpark, SQL, Shell Scripting, R
Visualization Tools: Power BI, Tableau, Cognos
Data Governance: Informatica MDM, AXON, EDC
Certifications Preferred:
Microsoft Certified: Azure Data Engineer Associate
Databricks Data Engineer Associate / Professional
Job Classification
Industry: IT Services & Consulting
Functional Area / Department: Data Science & Analytics
Role Category: Data Science & Machine Learning
Role: Data Science & Machine Learning - Other
Employement Type: Full time
Contact Details:
Company: Exponentia Datalabs
Location(s): Mumbai
Keyskills:
Azure
Databricks
Data Governance
Azure Data Lake