Data Engineer
- Programming.Com
- 8 - 12 years
- Pune
- 12 hours ago
- Email to a friend
- Report this job
Job Description
immediate joiners only
Senior Data Engineer (Airflow & PostgreSQL Specialist)
Role Overview
We are looking for a heavy-hitting Senior Data Engineer to lead the architecture and optimization of our data ecosystem. With a core focus on Apache Airflow and PostgreSQL, you will be responsible for building industrial-grade data pipelines that are scalable, self-healing, and highly performant. This is a role for an engineer who thrives on solving complex orchestration challenges and fine-tuning relational databases for high-volume analytical workloads.
Key Responsibilities
1. Advanced Airflow Orchestration (Core)
- Architect and manage complex Airflow DAGs, moving beyond basic scheduling to implement dynamic task generation and custom operators.
- Implement high-level dependency management, including Cross-DAG dependencies, sensors, and sophisticated retry/alerting logic.
- Optimize the Airflow environment (KubernetesExecutor/CeleryExecutor) for reliability and resource efficiency.
2. PostgreSQL Expert-Level Development
- Lead the design of PostgreSQL schemas, focusing on performance for both OLTP and OLAP patterns.
- Deep-dive into Query Optimization: Analyze EXPLAIN plans, implement advanced indexing strategies, and manage table partitioning.
- Ensure data integrity through advanced SQL techniques, stored procedures, and rigorous validation logic.
3. Scalable ETL/ELT & Big Data
- Develop robust pipelines using Python, Spark, and SQL to move data across Cloud Storage, Kafka, and Snowflake/Redshift.
- Design and maintain Data Lakes and Warehouses that support real-time and batch processing.
4. Data Governance & Reliability
- Implement Data Quality (DQ) frameworks (e.g., Great Expectations) to ensure accuracy and completeness.
- Establish data lineage, metadata management, and compliance standards (GDPR/FedRAMP).
- Monitor system health and implement automated recovery strategies for mission-critical pipelines.
Required Technical Skills
- Databases: Deep expertise in PostgreSQL (Performance tuning, Partitioning, Vacuuming). Proficiency in NoSQL (MongoDB/Cassandra) is a plus.
- Orchestration: Expert-level Apache Airflow (DAG design, Monitoring, Scaling).
- Languages: Advanced Python and SQL (Mandatory). Familiarity with Java or Scala.
- Big Data: Hands-on experience with Apache Spark and Kafka.
- Cloud: AWS (S3, Redshift, Glue) or Azure (ADLS, Synapse).
- DevOps: Strong understanding of Docker, Kubernetes, and CI/CD pipelines.
Qualifications
- Bachelors or Masters in Computer Science or a related technical field.
- 8+ years of experience in Data Engineering, with at least 3 years focused on high-scale Airflow/Postgres environments.
- Proven track record of architecting data systems from the ground up.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Programming.ComLocation(s): Pune
Keyskills: Airflow Postgresql Apache AWS Python Performance Optimization Big Data Kafka Ci/Cd Elt SQL GIT Docker Amazon Redshift Snowflake Tuning And Optimization Data Modeling Spark ETL Kubernetes
Similar positions
Programming.Com
Programming.com is a global technology services and software development company delivering high-quality digital solutions to enterprises and fast-growing businesses. We specialize in building scalable backend systems, modern web applications, and cloud-native platforms using cutting-edge technol...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in