Officer - Big Data Engineer - C11 - Hybrid
- Thriventree Consulting
- 3 to 7 Yrs
- Chennai
- 5 months ago
- Email to a friend
- Report this job
Job Description
- You will be responsible for designing, developing, and optimizing data processing solutions using a combination of Big Data technologies. Your focus will be on building scalable and efficient data pipelines for handling large datasets and enabling batch & real-time data streaming and processing. Your responsibilities will include developing Spark applications using Scala or Python (Pyspark) for data transformation, aggregation, and analysis. You will also need to develop and maintain Kafka-based data pipelines, which involves designing Kafka Streams, setting up Kafka Clusters, and ensuring efficient data flow. Additionally, you will create and optimize Spark applications using Scala and PySpark to process large datasets and implement data transformations and aggregations. Another important aspect of your role will be integrating Kafka with Spark for real-time processing. You will be building systems that ingest real-time data from Kafka and process it using Spark Streaming or Structured Streaming. Collaboration with data teams including data engineers, data scientists, and DevOps is essential to design and implement data solutions effectively. You will also need to tune and optimize Spark and Kafka clusters to ensure high performance, scalability, and efficiency of data processing workflows. Writing clean, functional, and optimized code while adhering to coding standards and best practices will be a key part of your daily tasks. Troubleshooting and resolving issues related to Kafka and Spark applications, as well as maintaining documentation for Kafka configurations, Spark jobs, and other processes are also important aspects of the role. Continuous learning and applying new advancements in functional programming, big data, and related technologies is crucial. Proficiency in the Hadoop ecosystem big data tech stack (HDFS, YARN, MapReduce, Hive, Impala), Spark (Scala, Python), Kafka, ETL processes, and data ingestion tools is required. Deep hands-on expertise in Pyspark, Scala, Kafka, programming languages such as Scala, Python, or Java for developing Spark applications, and SQL for data querying and analysis are necessary. Additionally, familiarity with data warehousing concepts, Linux/Unix operating systems, problem-solving, analytical skills, and version control systems will be beneficial in performing your duties effectively. This is a full-time position in the Technology job family group, specifically in Applications Development. If you require a reasonable accommodation to use search tools or apply for a career opportunity due to a disability, please review Accessibility at Citi. You can also refer to Citis EEO Policy Statement and the Know Your Rights poster for more information.,
Employement Category:
Employement Type: Full timeIndustry: IT Services & ConsultingRole Category: Not SpecifiedFunctional Area: Not SpecifiedRole/Responsibilies: Officer - Big Data Engineer - C11 - HybridContact Details:
Company: CitiLocation(s): Chennai+ View Contact
Keyskills: Scala Python Kafka Hadoop HDFS YARN MapReduce Hive Impala ETL SQL Data warehousing Analytical skills Problemsolving
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in