Senior Data Engineer
- Idexcel
- 4 - 6 years
- Hyderabad
- 2 days ago
- Email to a friend
- Report this job
Job Description
Databricks (Spark)
- Develop scalable ETL/ELT pipelines using PySpark (RDD/DataFrame APIs), Delta Lake, Auto Loader (cloudFiles), and Structured Streaming.
- Optimize jobs: partitioning, bucketing, Z-Ordering, OPTIMIZE + VACUUM, broadcast joins, AQE, checkpointing.
- Manage Unity Catalog: catalogs/schemas/tables, data lineage, permissions, secrets, tokens, and cluster policies.
- CI/CD for Databricks assets: notebooks, Jobs, Repos, MLflow
- Build Medallion Architecture (Bronze/Silver/Gold) with Delta Live Tables (DLT) and expectations for data quality.
- Event-driven ingestion: Kafka/Kinesis Databricks Streaming
Snowflake (DW & ELT)
- Model and implement star/snowflake schemas, data marts, and secure views.
- Performance tuning: clustering keys, micro-partitions, result caching, warehouse sizing, query profile
- Implement Task/Stream patterns for CDC; external tables for data lakes (S3); Snowpipe for near-real-time ingestion.
- Python/Snowpark for transformations and UDFs; SQL best practices (CTEs, window functions).
- Security: Row Level Security (RLS), Column Masking, OAuth/SCIM, network policies, data sharing (reader accounts).
AWS Data Engineering
- Storage & compute: S3 (lifecycle, encryption, partitioning), EMR (if needed), Lambda, Glue (ETL/Schema registry), Athena, Kinesis (Data Streams/Firehose), RDS/Aurora, Step Functions.
- Orchestration: MWAA/Airflow or Step Functions (error handling, retries, backfills, SLA alerts).
- Infra-as-code: Terraform/CloudFormation for reproducible environments (Databricks workspace, IAM, S3, networking).
- Security/compliance: IAM least privilege, KMS, VPC endpoints/private links, Secrets Manager, CloudTrail/CloudWatch, GuardDuty.
- Observability: CloudWatch metrics/logs, structured logging, datadog/Prometheus (optional), cost monitoring (tags/budgets).
Data Quality, Governance & Security
- Implement unit/integration tests for pipelines (e.g., pytest + Great Expectations + DLT expectations).
- Data contracts and schema evolution; monitor SLA/SLO; DQ dashboards (missingness, drift, freshness, completeness).
- PII handling: tokenization/pseudonymization, field-level encryption, KYB/KYC data flows adherence; audit trails.
- Cataloging & lineage through Unity Catalog and/or OpenLineage/Purview (if applicable).
DevOps & CI/CD
- Git workflows (branching, PR reviews), Databricks CLI/Terraform modules for jobs/clusters/UC, Snowflake DevOps (object versioning via schemachange or SQL-based migration).
- Automated testing in pipelines; feature flags, canary releases for data jobs; rollback strategies.
Client-Facing PoCs & Delivery
- Rapid PoC build: clearly defined success metrics, benchmark cost/performance, produce a transition plan to production.
- Present architectural decisions, trade-offs (Spark vs Snowflake ELT), and cost projections (Databricks DBU, Snowflake credits, storage egress).
- Produce runbooks, operational playbooks, and knowledge transfer documents for client teams.
Required Technical Skillset
- Databricks: PySpark, Delta Lake, Auto Loader, DLT, Jobs, Unity Catalog, MLflow basics.
- Snowflake: SQL, Snowpipe, Tasks/Streams, Snowpark (Python), warehouse sizing, performance tuning, security policies.
- Python: strong in packages for DE (pandas, pyarrow, pytest), robust error handling, typing, and packaging.
- Orchestration: Airflow DAGs (Sensors, Operators, XCom), Step Functions state machines.
- Streaming & CDC: Kafka/Kinesis, Debezium (nice-to-have), CDC patterns to Delta/Snowflake.
- AWS: S3, Glue, Lambda, Kinesis, IAM/KMS, VPC, CloudWatch; Terraform/CloudFormation.
- Data Modeling: 3NF/Dimensional, slowly changing dimensions (SCD Type 2), surrogate keys, surrogate vs natural debates.
- Security & Compliance: encryption at rest/in transit, tokenization, key rotation, audit logging, governance controls.
- Performance & Cost: Spark job tuning, Snowflake warehouse right-sizing, partitioning/clustering, object storage best practices.
Nice-to-Have:
- dbt (Snowflake) with tests & exposures; Great Expectations.
- Databricks SQL Warehouses and BI connectivity; Photon engine awareness.
- Lakehouse Federation (UC external locations); Delta Sharing; Iceberg
- Kafka Connect/Debezium, NiFi or MuleSoft (for data integrations).
- Experience in financial services
- Exposure to ISO/IEC 27001 controls in data platforms.
Education & Certifications
- Bachelors/Masters in CS/IT/EE or related.
- Certifications (plus): Databricks Data Engineer Associate/Professional, Snowflake SnowPro Core/Advanced, AWS Solutions Architect/Big Data/DP.
Job Classification
Industry: Recruitment / StaffingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: IdexcelLocation(s): Hyderabad+ View Contact
Keyskills: Data Engineering PySpark Auto Loader Data Quality DevOps Snowflake Delta Lake CI/CD Data Modeling Databricks ETL AWS Data Governance Python
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in