Data Engineer (Apache Flume & Kafka)
- Coforge
- 4 - 8 years
- Hyderabad
- 1 month ago
- Email to a friend
- Report this job
Job Description
We are seeking a skilled and detail-oriented Data Engineer (Apache Flume) to join our team. The ideal candidate will have 4+ years of hands-on experience in development, with a strong grasp of Apache Flume, data modelling, and migration. You will work closely with cross-functional teams to deliver data-driven solutions that empower strategic decision-making across the organization.
To lead the migration of existing logs from SQL Server databases to an Apache Flume application and establish a robust pipeline for daily log ingestion. The ideal candidate will have hands-on experience in data engineering, log management, and automation of ETL processes.
Key Responsibilities
Log Migration:
- Design and implement a solution to extract and transfer historical logs from SQL Server to Apache Flume.
- Ensure data integrity, completeness, and consistency during migration.
- Daily Log Ingestion:
- Develop and automate a pipeline to move newly generated logs from SQL Server to Flume on a daily basis.
- Monitor and maintain the pipeline for performance and reliability.
- System Integration & Optimization:
- Collaborate with infrastructure and application teams to ensure seamless integration.
- Optimize data flow and troubleshoot issues in real-time.
- Documentation & Reporting:
- Maintain detailed documentation of architecture, processes, and configurations.
- Provide regular updates and reports to stakeholders.
Required Skills & Qualifications:
- 3+ years of experience in data engineering or ETL development.
- Strong expertise in SQL Server and Apache Flume.
- Proficiency in scripting languages (e.g., Python, Shell) for automation.
- Experience with log management, data pipelines, and system integration.
- Familiarity with big data ecosystems (e.g., Hadoop, Kafka) is a plus.
- Excellent problem-solving and communication skills.
Share your resume over Aa***********a@co****e.com
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: ConsultingRole Category: IT ConsultingRole: IT Consulting - OtherEmployement Type: Full timeContact Details:
Company: CoforgeLocation(s): Hyderabad
Keyskills: Apache Flume Airflow Data Engineering Kafka Spark AWS SQL Python
Similar positions
Coforge
Coforge is a leading global IT solutions organization, enabling its clients to transform at the intersect of unparalleled domain expertise and emerging technologies to achieve real-world business impact. A focus on very select industries, a detailed understanding of the underlying processes of those...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in