Senior Software Engineer - Azure Databricks, SQL, Pyspark
- Infogain
- 6 - 8 years
- Noida, Gurugram
- 2 months ago
- Email to a friend
- Report this job
Job Description
Work Location: Bengaluru, Pune, Mumbai, Noida, Gurugram.
Notice Period: 0 to 15 Days
Key Responsibilities:
- Develop scalable data pipelines using Azure Data Factory (ADF), Databricks, PySpark, and Delta Lake to support ML and AI workloads.
- Optimize and transform large datasets for feature engineering, model training, and real-time AI inference.
- Build and maintain lakehouse architecture using Azure Data Lake Storage (ADLS) & Delta Lake.
- Work closely with ML engineers & Data Scientists to deliver high-quality, structured data for training Generative AI models.
- Implement MLOps best practices for continuous data processing, versioning, and model retraining workflows.
- Monitor & improve data quality using Azure Data Quality Services
- Ensure cost-efficient data processing in Databricks using Photon, Delta Caching, and Auto-Scaling Clusters.
- Secure data pipelines by implementing RBAC, encryption, and governance
Required Skills & Experience:
- 6+ years of experience in Data Engineering with Azure & Databricks.
- Proficiency in PySpark, SQL, and Delta Lake for large-scale data transformations.
- Strong experience with Azure Data Factory (ADF), Azure Synapse, and Event Hubs.
- Hands-on experience in building feature stores for ML models.
- Experience with ML model deployment and MLOps pipelines (MLflow, Kubernetes, or Azure ML) is a plus.
- Good understanding of Generative AI concepts and handling unstructured data
- Strong problem-solving, debugging, and performance optimization skills.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QA,
Role Category: Software Development
Role: Data Engineer
Employement Type: Full time
Contact Details:
Company: InfogainLocation(s): Noida, Gurugram
Keyskills: Pyspark Complex Sql Query Azure Databricks Azure Data Bricks SQL Query Optimization Sql Query Writing
Similar positions
Infogain
Infogain is a Silicon Valley headquartered company with software platform engineering and deep domain expertise in the travel, retail, insurance and high technology industries. We accelerate the delivery of digital customer engagement systems using digital technologies such as cloud, mic...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in