Senior - Azure Data Engineer ( data bricks with Python )
- Pylon Management
- 5 - 8 years
- Hyderabad
- 11 days ago
- Email to a friend
- Report this job
Job Description
to 8 years of experience in data engineering with at least 2+ years hands-on with Azure Databricks for pipeline development and data modelling.
Technical Skills:
Azure Ecosystem Expertise:
Azure Databricks: End-to-end development including Delta Lake, notebook workflows, and ML integrations.
Azure Data Factory: For orchestration and data pipeline integration.
Azure Storage (Blob/Data Lake Gen2): Working with storage accounts for structured and unstructured data.
Azure Key Vault: Securing secrets and managing credentials in pipelines.
Azure DevOps: CI/CD for data pipelines and notebooks.
Big Data & Distributed Processing:
Apache Spark (via Databricks): Advanced knowledge of Spark SQL, PySpark, DataFrames, RDDs, and performance tuning.
Delta Lake: Understanding ACID transactions, schema enforcement, time travel, and optimizing data formats.
Data Modelling and Warehousing:
Dimensional modelling (Star/Snowflake schemas)
Knowledge of modern data warehousing principles.
Experience implementing medallion architecture (Bronze/Silver/Gold layers).
Experience working with Parquet, JSON, CSV, or other data formats
Programming Languages:
Python: For data transformation, notebook development, automation.
SQL: Strong grasp of SQL for querying and performance tuning.
Scala (nice to have): Useful for certain Spark applications.
CI/CD and Infrastructure Automation:
Experience with Git repositories (branching, PRs).
Automated deployments via Azure DevOps Pipelines.
Data Engineering & Analytical Skills:
ETL/ELT pipeline design and optimization.
Data quality and validation frameworks.
Security & Governance:
RBAC and access controls within Azure and Databricks.
Data encryption, secure key management.
GDPR, HIPAA, or other compliance-aware data handling
Soft Skills & Leadership:
Stakeholder Communication: Translate technical jargon into business value.
Project Ownership: End-to-end delivery including design, implementation, and monitoring.
Mentorship: Guide junior engineers and establish best practices.
Agile Practices: Work in sprints, participate in scrum ceremonies, story estimation.
Education:
Bachelors or masters degree in computer science, Data Engineering, or a related field.
Certifications such as Azure Data Engineer, Databricks Certified Data Engineer Professional is a plus.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QA
Role Category: Software Development
Role: Data Engineer
Employement Type: Full time
Contact Details:
Company: Pylon ManagementLocation(s): Hyderabad
Keyskills: python Azure Databricks data engineer Adb SQL
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in