Databricks - Lead
- Iris Software
- 4 - 8 years
- Noida, Gurugram
- 14 hours ago
- Email to a friend
- Report this job
Job Description
Databricks - Lead- Apache Spark (Core, SQL, Structured Streaming)
- PySpark
- Databricks (AWS/Azure)
- Advanced SQL
- Jenkins, Git/GitHub/Bitbucket
- Python, SQL
- Java/Scala (Nice to have)
- AWS (S3, EMR, EC2, IAM, CloudWatch)/Azure- Preferred
- Databricks Runtime Cluster Management- Preferred
- Apache Kafka
- Snowflake integration
- Airflow
- Perform Spark performance tuning
- DAG analysis
- Shuffle optimization
- Partition strategies
- Executor/memory tuning
- Experience in financial services, regulatory reporting, or enterprise data platforms.
- Hands-on experience in Delta Lake optimization and incremental processing strategies.
- Experience with Snowflake data warehousing.
- Databricks Certification (Professional level preferred).
- Strong understanding of distributed computing principles.
- Design and develop scalable batch and near-real-time ETL/ELT pipelines using Databricks (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming).
- Modernize legacy SQL/Hive/stored procedure workflows into distributed Spark-native architectures.
- Perform Spark performance tuning, including:
- Build structured streaming pipelines using Kafka and Spark Structured Streaming.
- Design dimensional data models (Fact/Dimension, SCD Type 2).
- Orchestrate pipelines using Databricks Workflows / Apache Airflow.
- Integrate CI/CD pipelines using Jenkins, Git, Bitbucket/GitHub for automated deployment across DEV/UAT/PROD.
- Lead end-to-end solution design for data platforms using Databricks (batch, streaming, ML workloads)
- Define architecture patterns like Lakehouse, Medallion (Bronze/Silver/Gold)
- Act as SME for Databricks, Spark, and data engineering best practices
- Lead and mentor a team of data engineers (typically 5-10 members)
- Conduct code reviews, enforce best practices, and ensure delivery quality
- Guide team in troubleshooting complex technical issues
- Collaborate with business stakeholders, architects, and product owners
- Translate business requirements into technical designs and sprint tasks
- Drive technical decisions (performance vs cost vs scalability)
- Own end-to-end delivery of data projects
- Ensure adherence to Agile processes, SLAs, and governance models
- Performdesign reviews, estimations, and risk management
- Data Science and Machine Learning - Data Science and Machine Learning - Databricks
- Cloud - Azure - Azure Data Factory (ADF), Azure Databricks, Azure Data Lake Storage, Event Hubs, HDInsight
- Big Data - Big Data - Pyspark
- Database - Database Programming - SQL
- Data Science and Machine Learning - Data Science and Machine Learning - Python
- Beh - Communication and collaboration
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Technical LeadEmployement Type: Full timeContact Details:
Company: Iris SoftwareLocation(s): Noida, Gurugram
Keyskills: databricks hive scala sprint data warehousing bitbucket sql java git data science iam spark ec2 devops apache spark jenkins big data etl architecture azure s3 python github airflow machine learning kafka agile aws
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in