Data Engineer (Kafka & Databricks)
- Coforge
- 4 - 8 years
- Hyderabad
- 3 days ago
- Email to a friend
- Report this job
Job Description
We are seeking a skilled and detail-oriented Data Engineer (Apache Flume/Kafka) to join our team. The ideal candidate will have 4+ years of hands-on experience in development, with a strong grasp of Apache Flume, data modelling, and migration. You will work closely with cross-functional teams to deliver data-driven solutions that empower strategic decision-making across the organization.
To lead the migration of existing logs from SQL Server databases to an Apache Flume application and establish a robust pipeline for daily log ingestion. The ideal candidate will have hands-on experience in data engineering, log management, and automation of ETL processes.
Key Responsibilities
Log Migration:
- Design and implement a solution to extract and transfer historical logs from SQL Server to Apache Flume.
- Ensure data integrity, completeness, and consistency during migration.
- Daily Log Ingestion:
- Develop and automate a pipeline to move newly generated logs from SQL Server to Flume on a daily basis.
- Monitor and maintain the pipeline for performance and reliability.
- System Integration & Optimization:
- Collaborate with infrastructure and application teams to ensure seamless integration.
- Optimize data flow and troubleshoot issues in real-time.
- Documentation & Reporting:
- Maintain detailed documentation of architecture, processes, and configurations.
- Provide regular updates and reports to stakeholders.
Required Skills & Qualifications:
- 3+ years of experience in data engineering or ETL development.
- Strong expertise in SQL Server and Apache Flume.
- Proficiency in scripting languages (e.g., Python, Shell) for automation.
- Experience with log management, data pipelines, and system integration.
- Familiarity with big data ecosystems (e.g., Hadoop, Kafka) is a plus.
- Excellent problem-solving and communication skills.
Share your resume over Aa***********a@co****e.com
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: ConsultingRole Category: IT ConsultingRole: IT Consulting - OtherEmployement Type: Full timeContact Details:
Company: CoforgeLocation(s): Hyderabad
Keyskills: Apache Flume SQL Python Airflow Data Engineering Kafka Spark AWS Data Bricks
Similar positions
Coforge
Ernst & Young (EY) is a global professional services firm and one of the Big Four accounting firms. It offers audit, tax, consulting, and advisory services to clients worldwide, helping businesses with transformation, risk management, and innovation. With a presence in over 150 countries and nea...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in