Azure Data Engineer (Senior)
- Infogain
- 6 - 9 years
- Bengaluru
- 2 months ago
- Email to a friend
- Report this job
Job Description
Key Responsibilities
1. Data Quality Development & Monitoring
- Design and implement automated data quality rules and validation checks using Databricks (Delta Lake) and PySpark .
- Build and operationalize data quality workflows in Ataccama ONE / Ataccama Studio .
- Perform data profiling, anomaly detection, and reconciliation across systems and data sources.
- Establish thresholds, KPIs, and alerts for data quality metrics.
2. Root Cause Analysis & Issue Management
- Investigate data anomalies and quality incidents using SQL, Python, and Ataccama diagnostics.
- Collaborate with data engineers and business analysts to identify and remediate root causes.
- Document recurring data issues and contribute to preventive automation solutions.
3. Collaboration & Governance Support
- Partner with data stewards, governance, and analytics teams to define and maintain DQ rules and SLAs.
- Contribute to metadata enrichment, lineage documentation, and data catalog integration.
- Support adoption of DQ frameworks and promote data reliability best practices.
4. Automation & Continuous Improvement
- Integrate DQ validations into orchestration tools (Airflow, Databricks Workflows, or ADF).
- Leverage Python/Pyspark libraries to complement existing platforms.
- Propose process improvements to enhance automation, monitoring, and exception management.
Core Technical Skills
Category
Tools / Skills
Data Engineering & Quality
Databricks (Delta Lake), PySpark, SQL, Python
DQ Platforms
Ataccama ONE / Studio (DQ rules, workflows, profiling)
Orchestration
Apache Airflow, Azure Data Factory, or Databricks Jobs
Data Warehouses
Databricks Lakehouse
Version Control / CI-CD
Git, GitHub Actions, Azure DevOps
Data Catalog / Lineage (Optional)
Collibra, Alation, Ataccama Catalog
Cloud Environments
Azure (preferred), AWS, or GCP
Qualifications & Experience
- Bachelor s degree in Computer Science, Information Systems, Statistics, or related field.
- 6 9 years of experience in data quality, data engineering, or analytics operations.
- Strong command of SQL , Python , and PySpark for data validation and troubleshooting.
- Proven experience with Ataccama DQ rule creation and monitoring .
- Hands-on exposure to Databricks for building and running data pipelines.
- Working knowledge of reconciliation processes, data profiling, and DQ metrics.
Soft Skills & Attributes
- Analytical thinker with strong problem-solving abilities.
- Detail-oriented and methodical approach to troubleshooting.
- Strong communication skills for cross-functional collaboration.
- Proactive mindset, capable of owning issues through resolution.
- Comfortable balancing hands-on technical work with business stakeholder interaction.
Preferred / Nice to Have
- Exposure to data governance frameworks or MDM initiatives.
- Familiarity with observability tools (Grafana, Datadog, Prometheus).
- Understanding of CI/CD practices for data quality deployment.
- Certification in Databricks, Ataccama, or a major cloud platform (Azure/AWS).
Success Indicators
- Increase in automated data quality coverage across critical datasets.
- Reduction in recurring manual DQ exceptions.
- Improved timeliness and accuracy of data available for analytics.
- Positive stakeholder feedback on data trust and reliability.
- 6-8 Years
- Primary Skill: Data Engineering
- Sub Skill(s): Data Engineering
- Additional Skill(s): Python, databricks, SQL, Azure Data Factory
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: InfogainLocation(s): Bengaluru+ View Contact
Keyskills: Computer science Telecom Automation Analytical Reconciliation Healthcare Data quality Continuous improvement Analytics Monitoring
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Infogain
A Silicon-Valley headquartered company, Infogain is a global business oriented IT consulting provider of front-end, customer-facing technologies, processes and applications, leading to a more efficient and streamlined customer experience. We want our clients€™ interactions with their cus...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in