Design, develop, and optimize scalable data pipelines using Databricks (PySpark, Scala, SQL).
Implement ETL/ELT workflows for large-scale data integration across cloud and on-premise environments.
Leverage Microsoft Fabric (Data Factory, OneLake, Lakehouse, DirectLake, etc.) to build unified data solutions.
Collaborate with data architects, analysts, and stakeholders to deliver business-critical data models and pipelines.
Monitor and troubleshoot performance issues in data pipelines.
Ensure data governance, quality, and security across all data assets.
Work with Delta Lake, Unity Catalog, and other modern data lakehouse components.
Automate and orchestrate workflows using Azure Data Factory, Databricks Workflows, or Microsoft Fabric pipelines.
Participate in code reviews, CI/CD practices, and agile ceremonies.
Required Skills:
5-7 years of experience in data engineering, with strong exposure to Databricks .
Proficient in PySpark, SQL, and performance tuning of Spark jobs.
Hands-on experience with Microsoft Fabric components .
Experience with Azure Synapse, Data Factory, and Azure Data Lake.
Understanding of Lakehouse architecture and modern data mesh principles.
Familiarity with Power BI integration and semantic modeling (preferred).
Knowledge of DevOps, CI/CD for data pipelines (e.g., using GitHub Actions, Azure DevOps).
Excellent problem-solving, communication, and collaboration skills.
Job Classification
Industry: Management ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full time