Sr Data Engineer @ Lorven Technologies

Job Description :

We are seeking a highly skilled and motivated Data Engineer to join our growing data team. The ideal candidate will have a strong background in data processing, database management, and reporting, with expertise in building and maintaining scalable data pipelines. You will be instrumental in transforming raw data into valuable insights and ensuring data availability across the organization. Proficiency in SQL, Informatica, PySpark, and Tableau, along with excellent communication and collaboration skills, are essential to this role.

Key Responsibilities

• Design, develop, and maintain robust ETL/ELT pipelines using Informatica and PySpark.
• Migrate ETL processes from Informatica to PySpark for improved performance and scalability.
• Automate data workflows and orchestrate complex pipelines using Apache Airflow.
• Collaborate with data analysts, scientists, and business stakeholders to deliver actionable insights.
• Ensure high-quality, maintainable code using best practices including version control, unit testing, and CI/CD.
• Optimize data solutions for performance, scalability, and reliability in cloud or hybrid environments.
• Write complex SQL queries to extract, clean, and transform data for reporting and analytics.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime.
• Maintain comprehensive documentation for data processes, architectures, and pipeline workflows.
• Ensure data integrity, privacy, and security throughout the lifecycle.

Must-Have Qualifications

• 7+ years of hands-on experience in data engineering roles.
• Strong expertise with Informatica for ETL development.
• Advanced Python programming skills with experience in PySpark for big data processing.
• Proven experience with Apache Airflow for workflow orchestration.
• Solid understanding of data engineering principles and data modeling.
• Experience in building real-time/streaming data pipelines.
• Familiarity with data warehousing concepts.
• Excellent problem-solving skills and attention to detail.
• Strong verbal and written communication skills.
• Ability to work independently and manage multiple tasks in a fast-paced environment.
• Demonstrated ability to write well-documented, clean, and efficient code.

Good to Have

• Experience with cloud-based data platforms such as AWS, GCP, or Azure.
• Kubernetes experience for deploying and managing containerized applications.
• AWS Solution Architect certification.
• Familiarity with modern data warehousing tools and services.

Education

• BE/B.Tech/MS/M.Tech/ME in Computer Science, Engineering, or a related field from a reputed institute.