Databricks Engineer- CHRTN
- Puresoftware
- 5 - 10 years
- Bengaluru
- 4 months ago
- Email to a friend
- Report this job
Job Description
Design, build, and maintain scalable and efficient ETL/ELT pipelines in Azure Databricks to process structured, semi-structured, and unstructured insurance data from multiple internal and external sources
Collaborate with data architects, modellers, analysts, and business stakeholders to gather data requirements and deliver fit-for-purpose data assets that support analytics, regulatory, and operational needs
Develop, test, and optimize data transformation routines, batch and streaming solutions (leveraging tools such as Azure Data Factory, Data Lake Storage Gen2, Azure Event Hubs, and Kafka) to ensure timely and accurate data delivery
Implement rigorous data quality, validation, and cleansing procedures with a focus on enhancing reliability for high-stakes insurance use cases, reporting, and regulatory outputs
Integrate Informatica tools to facilitate data governance, including the capture of data lineage, metadata, and data cataloguing as required by regulatory and business frameworks
Ensure robust data security by following best practices for RBAC, managed identities, encryption, and compliance with Hong Kongs PDPO, GDPR, and other relevant regulatory requirements
Automate and maintain deployment pipelines using GitHub Actions to ensure efficient, repeatable, and auditable data workflows and code releases
Conduct root cause analysis, troubleshoot pipeline failures, and proactively identify and resolve data quality or performance issues
Produce and maintain comprehensive technical documentation for pipelines, transformation rules, and operational procedures to ensure transparency, reuse, and compliance
Apply subject matter expertise in Hong Kong Life and General Insurance to ensure that development captures local business needs and industry-specific standards
Job Classification
Industry: Management ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Quality Assurance and TestingRole: Blockchain Quality Assurance EngineerEmployement Type: Full timeContact Details:
Company: PuresoftwareLocation(s): Bengaluru
Keyskills: metadata github data security data governance Data quality Informatica Subject matter expertise Operations Analytics Technical documentation
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in