Data Platform Engineer (GCP Focus)
- Infinite
- 5 - 10 years
- Bengaluru
- 2 months ago
- Email to a friend
- Report this job
Job Description
Role : Data Platform Engineer (GCP Focus) Key ResponsibilitiesData Pipeline Automation & Engineering
- Design, build, and automate scalable, production-grade data pipelines on GCP using core services such as Cloud Composer (Airflow) , Dataflow (Apache Beam) , and BigQuery .
- Develop and implement continuous integration and continuous deployment (CI/CD) workflows for data processing and analytics pipelines using tools like Cloud Build , GitHub Actions , or Jenkins .
- Implement reusable data frameworks, templates, and libraries to standardize pipeline deployments, configuration management, and promote "Infrastructure as Code" principles.
- Orchestrate complex, multi-stage ETL/ELT pipelines across diverse data sources and environments, ensuring efficient resource utilization and low latency.
- Implement and manage automated data validation, schema checks, and anomaly detection using industry-leading tools like Great Expectations , dbt tests , or custom Python frameworks.
- Integrate quality gates directly into CI/CD workflows to ensure early issue detection and continuously improve overall data reliability (Data Reliability Engineering - DRE principles).
- Schedule, monitor, and optimize data workflows, ensuring strict adherence to data delivery SLAs.
- Set up and maintain proactive monitoring, logging, and automated alerting for all data pipelines and platform components using Stackdriver (Cloud Monitoring & Logging) , Prometheus , or Grafana .
- Develop and maintain comprehensive dashboards to track critical metrics, including data health, SLA adherence, and pipeline operational performance.
- Integrate and manage data assets, schemas, and metadata within Google Data Catalog or equivalent metadata management platforms.
- Enforce robust governance policies, including data lineage tracking, strict access control (IAM), and compliance standards for sensitive data.
- 5+ years of professional experience in data engineering, data platform operations, or a similar cloud-native technical role.
- Strong expertise in the Google Cloud Platform (GCP) data stack: BigQuery, Dataflow, Cloud Composer, Pub/Sub, Cloud Functions, and Cloud Build.
- High proficiency in Python , SQL , and general automation scripting.
- Hands-on experience with CI/CD principles and tools, including GitOps and Infrastructure as Code (IaC) using Terraform or Cloud Deployment Manager .
- Proven experience with data quality and testing frameworks such as Great Expectations, dbt, or PyTest.
- Working knowledge of observability, logging, and monitoring frameworks for high-volume data systems.
- Familiarity with metadata management, data lineage tools, and establishing data governance policies.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: InfiniteLocation(s): Bengaluru
Keyskills: Automation Data validation Configuration management Schema Deployment Data processing Data quality Operations Analytics SQL
Similar positions
Infinite
About Xperience Infinite: Xperience Infinite specializes in developing cutting-edge Computerized Maintenance Management System (CMMS) solutions tailored for UK-based retail giants. Our innovative products are designed to manage and maintain assets and resources efficiently, ensuring optimal perfor...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in