Data Engineer
- GSPANN
- 5 - 10 years
- Hyderabad
- 22 days ago
- Email to a friend
- Report this job
Job Description
Role Overview
We are looking for a highly skilled Data Engineer to join our data engineering team. The ideal candidate will have hands-on experience in building and maintaining data pipelines using Azure Databricks, Azure Data Factory, PySpark, and Microsoft Fabric (OneLake, Lakehouse, Dataflows, etc.). You will be responsible for designing robust data architectures, optimizing data workflows, and ensuring data quality and governance.
Key Responsibilities
- Design, build, and maintain scalable data pipelines using Azure Data Factory (ADF) and Microsoft Fabric Pipelines / Dataflows.
- Develop data transformations using PySpark / Spark SQL in Azure Databricks or Microsoft Fabric Notebooks.
- Build and maintain Lakehouse architecture on Microsoft Fabric (OneLake), following medallion (bronze / silver / gold) patterns or as required.
- Model data (dimensional modeling, star schema, etc.) for analytical use cases, data warehousing, and BI.
- Optimize performance of Spark jobs, data pipelines, and storage (partitioning, caching, indexing).
- Monitor, troubleshoot, and tune data pipelines for reliability, latency, and cost.
- Implement and enforce data quality practices, data validation, and data lineage.
- Collaborate with data analysts, data scientists, and business stakeholders to understand and translate data requirements.
- Ensure data governance, security, and access control in Microsoft Fabric, including row-level / column-level security where relevant.
- Automate deployments and orchestrate CI/CD for data pipelines and notebook code, using tools like Azure DevOps / Git.
- Document data architecture, pipeline design, data flows, and best practices.
- Stay up-to-date with new features in Azure Databricks, ADF, and Microsoft Fabric, and drive adoption of best practices.
Required Skills & Experience
- Experience: 38+ years (depending on seniority) in data engineering or a related role.
- Cloud: Strong experience with Azure, specifically Azure Data Factory, Databricks, and other Azure data services.
- Big Data / Spark: Proficient in PySpark (or Spark), Spark SQL, familiarity with Delta Lake or similar.
- Microsoft Fabric: Experience working with Fabrics Lakehouse, Dataflows, Pipelines, Notebooks, or other Fabric components.
- Note: Microsoft Fabric is a unified data platform integrating data movement, OneLake, Spark, etc. prepzeee.wixsite.com+2unitedtechno.com+2
- Programming: Strong skills in Python, especially for data transformation and scripting.
- SQL: Deep understanding of SQL, relational data modeling, data warehousing principles.
- Data Modeling: Experience in dimensional modeling, star / snowflake schemas, or medallion architecture.
- DevOps / Automation: Experience with version control (Git), CI/CD pipelines for data workloads, and scheduling.
- Data Governance: Understanding of data governance, lineage, security, and compliance within data platforms.
- Collaboration: Good communication skills to work with cross-functional teams (BI, Data Science, Business).
- Problem-Solving: Ability to debug complex data issues, optimize performance, and improve pipeline resiliency.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Production, Manufacturing & EngineeringRole Category: EngineeringRole: Engineering - OtherEmployement Type: Full timeContact Details:
Company: GSPANNLocation(s): Hyderabad
Keyskills: Data Engineering Azure Data Factory Pyspark Microsoft Fabric Snowflake Azure Databricks SQL
Similar positions
GSPANN
Why Choose GSPANN? We are a close-knit family of 1900+ members who support one another and celebrate each others successes. We work together, socialize together, and actively serve the communities we live in. With the passion to explore solutions to the most challenging business problems, we suppo...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in