Lead Data Engineer
- Atyeti
- 8 - 11 years
- Pune
- 4 months ago
- Email to a friend
- Report this job
Job Description
Role & responsibilities
- Develop and Maintain Data Pipelines: Design, develop, and manage scalable ETL pipelines to process large datasets using PySpark, Databricks, and other big data technologies.
- Data Integration and Transformation: Work with various structured and unstructured data sources to build efficient data workflows and integrate them into a central data warehouse.
- Collaborate with Data Scientists & Analysts: Work closely with the data science and business intelligence teams to ensure the right data is available for advanced analytics, machine learning, and reporting.
- Optimize Performance: Optimize and tune data pipelines and ETL processes to improve data throughput and reduce latency, ensuring timely delivery of high-quality data.
- Automation and Monitoring: Implement automated workflows and monitoring tools to ensure data pipelines are running smoothly, and issues are proactively addressed.
- Ensure Data Quality: Build and maintain validation mechanisms to ensure the accuracy and consistency of the data.
- Data Storage and Access: Work with data storage solutions (e.g., Azure, AWS, Google Cloud) to ensure effective data storage and fast access for downstream users.
- Documentation and Reporting: Maintain proper documentation for all data processes and architectures to facilitate easier understanding and onboarding of new team members.
Skills and Qualifications:
- Experience: 5+ years of experience as a Data Engineer or similar role, with hands-on experience in designing, building, and maintaining ETL pipelines.
- Technologies:
- Proficient in PySpark for large-scale data processing.
- Strong programming experience in Python, particularly for data engineering tasks.
- Experience working with Databricks for big data processing and collaboration.
- Hands-on experience with data storage solutions (e.g., AWS S3, Azure Data Lake, or Google Cloud Storage).
- Solid understanding of ETL concepts, tools, and best practices.
- Familiarity with SQL for querying and manipulating data in relational databases.
- Experience working with data orchestration tools such as Apache Airflow or Luigi is a plus.
- Data Modeling & Warehousing:
- Experience with data warehousing concepts and technologies (e.g., Redshift, Snowflake, or BigQuery).
- Knowledge of data modeling, data transformations, and dimensional modeling.
- Soft Skills:
- Strong analytical and problem-solving skills.
- Excellent communication skills, capable of explaining complex data processes to non-technical stakeholders.
- Ability to work in a fast-paced, collaborative environment and manage multiple priorities.
Preferred Qualifications:
- Bachelor's or Masters degree in Computer Science, Engineering, or a related field.
- Certification or experience with cloud platforms like AWS, Azure, or Google Cloud.
- Experience in Apache Kafka or other stream-processing technologies.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: AtyetiLocation(s): Pune+ View Contact
Keyskills: Pyspark Spark Data Bricks Python azure
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in