Azure Data Engineer (Lead)
- Infogain
- 9 - 12 years
- Bengaluru
- 5 months ago
- Email to a friend
- Report this job
Job Description
Key Responsibilities
1. Data Quality Framework Design & Leadership
- Define and implement enterprise-wide data quality frameworks and governance standards.
- Architect automated DQ pipelines using Databricks (Delta Lake) , PySpark , and Ataccama ONE .
- Design DQ monitoring architecture profiling, lineage integration, and alerting mechanisms.
- Establish KPIs and DQ scorecards to measure and communicate data trust metrics across domains.
2. Advanced Data Quality Development & Automation
- Build and optimize complex validation, reconciliation, and anomaly detection workflows using PySpark and Python.
- Implement rule-based and ML-based DQ checks, leveraging Ataccama workflows and open-source frameworks.
- Integrate DQ rules into CI/CD and orchestration platforms (Airflow, ADF, or Databricks Workflows).
- Partner with data engineers to embed DQ checks into ingestion and transformation pipelines.
3. Root Cause Analysis & Continuous Improvement
- Lead root-cause investigations for recurring DQ issues and drive long-term remediation solutions.
- Create and enforce best practices for rule versioning, DQ exception handling, and reporting.
- Own the playbook for DQ incident response and continuous optimization.
4. Stakeholder Management & Governance
- Act as the primary liaison between business data owners, IT, and governance teams.
- Translate business DQ requirements into technical implementation strategies.
- Drive executive-level reporting on DQ KPIs, SLAs, and issue trends.
- Contribute to metadata management, lineage documentation, and master data alignment.
5. Mentorship & Leadership
- Guide junior analysts and data engineers in developing robust DQ solutions.
- Lead cross-functional squads to implement new data quality capabilities or upgrades.
- Contribute to capability uplift training peers on DQ best practices, tools, and technologies.
Core Technical Skills
Category
Tools / Skills
Data Engineering & Quality
Databricks (Delta Lake), PySpark, SQL, Python
DQ Platforms
Ataccama ONE / Studio (rule authoring, workflow automation, profiling)
Orchestration & CI/CD
Apache Airflow, Azure Data Factory, Databricks Workflows, GitHub Actions
Data Warehouses
Databricks Lakehouse
Cloud & Infrastructure
Azure (preferred), AWS, or GCP; familiarity with Terraform or IaC concepts
Version Control / CI-CD
Git, GitHub Actions, Azure DevOps
Metadata & Governance
Collibra, Alation, Ataccama Catalog, OpenLineage
Monitoring & Observability
Grafana, Datadog, Prometheus for DQ metrics and alerts
Qualifications & Experience
- Bachelor s or Master s in Computer Science, Information Systems, Statistics, or related field.
- 9 12 years of experience in data quality, data engineering, or governance-focused roles.
- Proven experience designing and deploying enterprise DQ frameworks and automated checks.
- Strong expertise in Databricks , PySpark , and Ataccama for data profiling and rule execution.
- Advanced proficiency in SQL and Python for large-scale data analysis and validation.
- Solid understanding of data models, lineage, reconciliation, and governance frameworks
- Experience integrating DQ checks into CI/CD pipelines and orchestrated data flows.
Soft Skills & Leadership Attributes
- Strong analytical thinking and systems-level problem solving.
- Excellent communication and presentation skills for senior stakeholders.
- Ability to balance detail orientation with strategic vision.
- Influencer with a proactive, ownership-driven mindset.
- Comfortable leading cross-functional teams in fast-paced, cloud-native environments.
Preferred / Nice to Have
- Experience in financial, manufacturing, or large enterprise data environments.
- Familiarity with MDM , reference data , and data stewardship processes.
- Exposure to machine learning-driven anomaly detection or predictive data quality .
- Certifications: Databricks, Ataccama, or Cloud Data Engineering certifications (Azure/AWS).
Success Indicators
- Increased DQ rule coverage and automation across key data domains.
- Reduced manual DQ exceptions and faster remediation cycle times.
- Measurable improvement in data trust metrics and reporting accuracy.
- High stakeholder satisfaction with data availability and reliability.
- 8-11 Years
- Primary Skill: Data Engineering
- Sub Skill(s): Data Engineering
- Additional Skill(s): Python, databricks, SQL, Azure Data Factory
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: InfogainLocation(s): Bengaluru+ View Contact
Keyskills: Computer science Telecom Data analysis Automation Managed services Analytical Reconciliation Healthcare Data quality Monitoring
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Infogain
Infogain is a Silicon Valley headquartered company with software platform engineering and deep domain expertise in the travel, retail, insurance and high technology industries. We accelerate the delivery of digital customer engagement systems using digital technologies such as cloud, mic...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in