Pyspark Developer
- Capgemini Technology
- 5 Positions
- Bengaluru
- 4 days ago
- Email to a friend
- Report this job
Job Description
- Job Title: Pyspark Developer Grade: B2/C1 Job Description: We are looking for a skilled PySpark Developer with experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to join our team. The ideal candidate will play a crucial role in designing, developing, and implementing data solutions using PySpark for large-scale data processing and analytics. Responsibilities: Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation. Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources. Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights. Optimize PySpark jobs and data pipelines for performance, scalability, and reliability. Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows. Ensure data quality and integrity throughout all stages of data processing. Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation. Troubleshoot and debug issues related to data pipelines and processing. Ensure compliance with regulatory requirements and industry standards in all data processing activities. Implement best practices for data security, compliance, and privacy within Azure environment. Document technical specifications, data flows, and solution architecture. Requirements: Bachelors degree in Computer Science, Engineering, or a related field; Masters degree preferred. Proven experience as a PySpark Developer or similar role with a strong understanding of Apache Spark internals. Hands-on experience with Azure Databricks (ADB) and Azure Data Factory (ADF). Proficiency in Python programming language and solid understanding of SQL. Experience designing and optimizing data pipelines for ETL/ELT processes. Familiarity with cloud platforms, preferably Microsoft Azure. Excellent problem-solving skills and ability to think critically. Strong communication skills with the ability to collaborate effectively in a team environment. Experience in Financial, Risk, Compliance, or Banking domains is a plus. Experience identifying and mitigating financial risks in data processes. Ability to analyse data for potential risk factors and develop strategies to minimize financial risk. Ensure all data processes comply with relevant regulatory requirements and industry standards. Preferred Qualifications: Certification in Azure Data Engineering or related field. Knowledge of other big data technologies such as Hadoop, Hive, or Kafka. Familiarity with machine learning frameworks and techniques.
Employement Category:
Employement Type: Full timeIndustry: IT Services & Consulting
Role Category: Others
Functional Area: Not Specified
Role/Responsibilies: Pyspark Developer
Contact Details:
Company: Capgemini TechnologyLocation(s): Bengaluru
+ View Contact
Fraud Alert to job seekers!
₹ 4 to 9 Yr
Similar positions
Capgemini Technology
Capgemini Technology Services India Limited Capgemini in India is over 85,000 people strong across nine cities (Mumbai, Bangalore, Gurgaon, Noida, Gandhinagar, Hyderabad, Pune, Kolkata and Chennai - Trichy and Salem). A pioneer in the IT industry, Capgemini has over 45 years of global expertise ...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in