pyspark developer @ Kotak Mahindra Bank

Job Description

We are seeking a highly skilled Big Data Engineer with expertise in PySpark to join our dynamic team. As a Big Data Engineer, you will be responsible for designing, developing, and maintaining our big data infrastructure, as well as optimizing data processing pipelines. The ideal candidate will have a strong background in Python, PySpark, and big data technologies, along with experience in building scalable and efficient data pipelines.

Responsibilities:

1. Designing, implementing, and maintaining big data pipelines using PySpark.
2. Developing and optimizing ETL processes to ingest, process, and transform large volumes of data.
3. Collaborating with data scientists and analysts to understand their requirements and translate them into technical solutions.
4. Performance tuning of Spark jobs to improve efficiency and reduce processing time.
5. Troubleshooting and debugging issues in data pipelines and Spark applications.
6. Working with cross-functional teams to integrate new data sources and ensure data quality and integrity.
7. Implementing monitoring and alerting solutions to ensure the reliability and availability of data pipelines.
8. Staying up-to-date with the latest advancements in big data technologies and recommending improvements to existing systems.

Employement Category:

+ View Contact

Keyskills:   data python pyspark big