Data Engineer- Informatica- Pyspark
- InfoCepts
- 5 - 8 years
- Kolkata
- 9 months ago
- Email to a friend
- Report this job
Job Description
Develop and Implement Data Integration Workflows: Use ETL Tools/PySpark programming to create data integration workflows for data in traditional databases and data lake. Configure data mappings, transformations, and validations to ensure accurate and timely data integration across various systems and platforms.
Seamless Integration with Cloud-Based Applications: Implement seamless integration between Informatica (or IICS) and other cloud-based applications and services (ADLS, Cloudera etc.)
Support and Collaboration: Collaborate with Data Engineers and support engineers to design and develop solutions that meet requirements. Provide support before, during, and after deployment, addressing any issues that arise .
Performance Optimization: Assist with Level 2 and Level 3 application production issues, resolving challenging problems. Identify and implement tuning opportunities to improve overall system performance.
Service Excellence and SLA Commitments: Ensure service excellence by meeting service level agreement (SLA) commitments related to data integration.
Essential Skills:
- Strong expertise in PySpark and Informatica PowerCenter
- Experience in developing ETL pipelines using PySpark for processing large datasets
- In-depth knowledge of data integration patterns, database design, normalization, indexing and ETL/ELT processes
- Hands on experience in writing complex SQL queries, stored procedures, functions, and triggers to support business requirements
- Work with Hadoop, Hive, HDFS, and Delta Lake for data storage and retrieval
- Optimize Spark jobs for performance, scalability, and efficiency
- Implement data transformations, aggregations, and data quality checks in PySpark
- Integrate PySpark solutions with any cloud - AWS (Glue, EMR, S3), Azure Databricks, or GCP
- Monitor and troubleshoot Spark performance bottlenecks
- Experience CI/CD pipelines for automated code deployment
- Well versed in Python programming and Data Warehousing concepts
Desirable Skills:
- Cloud Data Integration and other relevant Informatica products
- Experience with data modelling, data warehousing, and database technologies.
- Exposure/experience with data modelling tool like SAS
- Proficiency in SQL, scripting languages, and API integrations
- Basis understanding of Power BI tool
- Certification on Informatica Power Centre or any other Informatica product suite
- Certification on Spark programming
- Good domain understanding on Banking
- Experience in implementing Feature Store
Qualifications:
- Must have minimum of 5 years of overall IT experience and 3+ years working experience in Informatica and PySpark (Data Integration)
- Bachelor s in engineering from a reputed Institute and prior experience on Projects in Data and Analytics Industry
Qualities:
- Strong communication and collaboration skills to work effectively with cross functional teams and stakeholders
- Ability to set, track, achieve and report on short/long term tasks
- Self-motivated and highly disciplined and organized
- Good people skills (will interface with people at varied skill and seniority levels)
Job Classification
Industry: Management ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: InfoCeptsLocation(s): Kolkata+ View Contact
Keyskills: Service level SAS Database design GCP Cloud Data quality Informatica Stored procedures Analytics Python
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
InfoCepts
Part of the global G4S security conglomerate, this Delhi-based entity has been operating since 1996. It provides business services such as facilities management, staffing for corporate and administrative roles, and security solutions. It reported revenue of around 31.4 crore in FY 2022 and functions...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in