Data Engineer
- Blend360 India
- 0 - 3 years
- Hyderabad
- 3 months ago
- Email to a friend
- Report this job
Job Description
- We are implementing a Media Mix Optimization (MMO) platform designed to analyze and optimize marketing investments across multiple channels
- This initiative requires a robust on-premises data infrastructure to support distributed computing, large-scale data ingestion, and advanced analytics
- The Data Engineer will be responsible for building and maintaining resilient pipelines and data systems that feed into MMO models, ensuring data quality, governance, and availability for Data Science and BI teams
- The environment integrates HDFS for distributed storage, Apache NiFi for orchestration, Hive and PySpark for distributed processing, and Postgres for structured data management
- This role is central to enabling seamless integration of massive datasets from disparate sources (media, campaign, transaction, customer interaction, etc.), standardizing data, and providing reliable foundations for advanced econometric modeling and insights.
Responsibilities:
Data Pipeline Development & Orchestration
o Design, build, and optimize scalable data pipelines in Apache NiFi to
o Design, build, and optimize scalable data pipelines in Apache NiFi to
automate ingestion, cleansing, and enrichment from structured, semi-structured, and unstructured sources.
Ensure pipelines meet low-latency and high-throughput requirements for distributed processing.
Data Storage & Processing
o Architect and manage datasets on HDFS to support high-volume,
o Architect and manage datasets on HDFS to support high-volume,
fault-tolerant storage.
o Develop distributed processing workflows in PySpark and Hive to
o Develop distributed processing workflows in PySpark and Hive to
handle large-scale transformations, aggregations, and joins across
petabyte-level datasets.
o Implement partitioning, bucketing, and indexing strategies to
o Implement partitioning, bucketing, and indexing strategies to
optimize query performance.
Database Engineering & Management
o Maintain and tune Postgres databases for high availability, integrity,
o Maintain and tune Postgres databases for high availability, integrity,
and performance.
o Write advanced SQL queries for ETL, analysis, and integration with
o Write advanced SQL queries for ETL, analysis, and integration with
downstream BI/analytics systems.
Collaboration & Integration
o Partner with Data Scientists to deliver clean, reliable datasets for
o Partner with Data Scientists to deliver clean, reliable datasets for
model training and MMO analysis.
o Work with BI engineers to ensure data pipelines align with reporting
o Work with BI engineers to ensure data pipelines align with reporting
and visualization requirements.
Monitoring & Reliability Engineering
o Implement monitoring, logging, and alerting frameworks to track data pipeline health.
o Troubleshoot and resolve issues in ingestion, transformations, and
o Implement monitoring, logging, and alerting frameworks to track data pipeline health.
o Troubleshoot and resolve issues in ingestion, transformations, and
distributed jobs.
Data Governance & Compliance
o Enforce standards for data quality, lineage, and security across systems.
o Ensure compliance with internal governance and external regulations.
o Enforce standards for data quality, lineage, and security across systems.
o Ensure compliance with internal governance and external regulations.
Documentation & Knowledge Transfer
o Develop and maintain comprehensive technical documentation for
o Develop and maintain comprehensive technical documentation for
pipelines, data models, and workflows.
o Provide knowledge sharing and onboarding support for cross-
o Provide knowledge sharing and onboarding support for cross-
functional teams.
- Bachelor s degree in Computer Science, Information Technology, or related field (Master s preferred).
- Proven experience as a Data Engineer with expertise in HDFS, Apach
Job Classification
Industry: Advertising & MarketingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Blend360 IndiaLocation(s): Hyderabad+ View Contact
Keyskills: Performance tuning Automation Debugging Agile Apache Information technology Monitoring SQL Python
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Blend360 India
BLEND360 is an award-winning, new breed Data Science Solutions Company focused on powering exceptional results to our Fortune 500/1000 clients and other organizations. We are a hyper growth company - born at the intersection of advanced analytics, data and technology. ...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in