Data Engineer
- Uplers
- 4 - 6 years
- Bengaluru
- 5 months ago
- Email to a friend
- Report this job
Job Description
Reference Code: HR1175440031573725
Designation: Data Engineer Azure & DatabricksJob Description for Data Engineer Azure & Databricks
Location: Bengaluru [Hybrid]
Experience Required: 4 to 6 years
Employment Type: Full-time
Department: Data Engineering / Analytics
(*Note: This is a requirement for one of Uplers Clients)About the Client: Our client is building scalable data platforms leveraging modern cloud-native and open-source technologies to drive decision intelligence across the organization. Our data lake is designed on the Microsoft Medallion Architecture, powered by Databricks and Azure-native services. We are looking for a passionate Data Engineer to join our team and help in expanding, optimizing, and maintaining our enterprise-grade data pipelines and architectures.Key Responsibilities
- Design, develop, and maintain robust ETL/ELT pipelines using Azure Data Factory (ADF), Databricks (PySpark/Scala), and SQL Server.
- Work on the Bronze-Silver-Gold (Medallion) architecture to organize data ingestion, cleansing, and transformation layers efficiently.
- Ingest and process both structured and unstructured data from diverse data sources such as APIs, cloud storages, transactional systems, and logs.
- Collaborate with data scientists, analysts, and business teams to ensure data availability and reliability for downstream use cases.
- Implement CI/CD for data pipelines using DevOps best practices (Azure DevOps / GitHub Actions).
- Optimize data processing performance on Databricks using Delta Lake, caching, Z-Ordering, and cluster tuning.
- Ensure data quality, observability, and lineage through tools like Great Expectations, OpenLineage, and dbt.
- Support metadata management and documentation for governance and compliance using tools like Apache Atlas or Unity Catalog.
- Explore and integrate open-source data tooling to supplement cloud-native workflows.
Technical Skills Required (Area-wise breakdown of tools and technologies)
- Cloud Platform - Azure (Data Factory, Data Lake Gen2, Blob Storage, Azure SQL DB, Key Vault, Azure DevOps)
- Compute & Processing - Databricks (Spark, PySpark, Delta Lake), SQL Server
- Orchestration - Azure Data Factory, dbt, Airflow (nice to have)
- Data Modeling - Dimensional Modeling, Data Vault (plus), Star/Snowflake schemas
- Programming Languages - Python, SQL, Spark (PySpark/Scala)
- CI/CD & DevOps - Azure DevOps, Git, GitHub Actions
- Data Observability & Quality - Great Expectations, Soda SQL, Monte Carlo (plus)
- Metadata Management - Unity Catalog, Apache Atlas
- Monitoring & Logging - Datadog, Azure Monitor, Log Analytics
- Open-Source Tools (Plus) - dbt, Apache Kafka, Apache Airflow, OpenLineage
Qualifications
- Bachelor's or Masters degree in Computer Science, Information Systems, or a related field.
- 46 years of hands-on experience in data engineering with strong exposure to Azure and Databricks ecosystems.
- Experience working in a modern data stack and familiarity with open-source best practices.
- Strong understanding of data warehouse concepts, ETL frameworks, and cloud-native data pipelines.
- Familiarity with data governance, lineage, cataloging, and access control.
- Strong problem-solving skills, attention to detail, and ability to work in agile, fast-paced environments.
Good to Have
- Certification in Azure Data Engineer Associate or Databricks Data Engineer Associate/Professional.
- Exposure to Data Mesh, Lakehouse architectures, and streaming systems like Kafka or Azure Event Hubs.
- Familiarity with machine learning pipeline orchestration (e.g., MLflow, Feature Store).
Job Classification
Industry: Software ProductFunctional Area / Department: Data Science & AnalyticsRole Category: Data Science & Machine LearningRole: Data EngineerEmployement Type: Full timeContact Details:
Company: UplersLocation(s): Bengaluru
Keyskills: Data Engineering Azure Databricks Pyspark Open Source Oracle ADF SQL Server Data Bricks SQL Azure Data Factory Ci / Cd Tools Delta Lake Azure Data Lake Data Lake Data Modeling Spark Python
Similar positions
Uplers
Mavlers is a full-service digital marketing agency that has propelled growth for over 7,000 brands and agencies worldwide. As Google, Mailchimp, WP VIP, Microsoft, Salesforce, and HubSpot partners, we possess the expertise to deliver high-impact projects and campaigns tailored to our clients uniqu...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in