Technical Lead - Sr. Data Engineer
- Edgematics Consulting
- 8 - 13 years
- Pune
- 15 days ago
- Email to a friend
- Report this job
Job Description
About This Role :
We are looking for a talented and experienced Data Engineer with Tech Lead with hands-on expertise in any ETL Tool with full knowledge about CI/CD practices with leading a team technically more than 5 and client facing and create Data Engineering, Data Quality frameworks. As a tech lead must ensure to build ETL jobs, Data Quality Jobs, Big Data Jobs performed performance optimization by understanding the requirements, create re-usable assets and able to perform production deployment and preferably worked in DWH appliances Snowflake / redshift / Synapse
Responsibilities
- Work with a team of engineers in designing, developing, and maintaining scalable and efficient data solutions using Any Data Integration (any ETL tool like Talend / Informatica) and any Big Data technologies.
- Design, develop, and maintain end-to-end data pipelines using Any ETL Data Integration (any ETL tool like Talend / Informatica) to ingest, process, and transform large volumes of data from heterogeneous sources.
- Have good experience in designing cloud pipelines using Azure Data Factory or AWS Glues/Lambda.
- Implemented Data Integration end to end with any ETL technologies.
- Implement database solutions for storing, processing, and querying large volumes of structured and unstructured and semi-structured data
- Implement Job Migrations of ETL Jobs from Older versions to New versions.
- Implement and write advanced SQL scripts in SQL Database at medium to expert level.
- Work with technical team with client and provide guidance during technical challenges.
- Integrate and optimize data flows between various databases, data warehouses, and Big Data platforms.
- Collaborate with cross-functional teams to gather data requirements and translate them into scalable and efficient data solutions.
- Optimize ETL, Data Load performance, scalability, and cost-effectiveness through optimization techniques.
- Interact with Client on a daily basis and provide technical progress and respond to technical questions.
- Implement best practices for data integration.
- Implement complex ETL data pipelines or similar frameworks to process and analyze massive datasets.
- Ensure data quality, reliability, and security across all stages of the data pipeline.
- Troubleshoot and debug data-related issues in production systems and provide timely resolution.
- Stay current with emerging technologies and industry trends in data engineering technologies, CI/CD, and incorporate them into our data architecture and processes.
- Optimize data processing workflows and infrastructure for performance, scalability, and cost-effectiveness.
- Provide technical guidance and foster a culture of continuous learning and improvement.
- Implement and automate CI/CD pipelines for data engineering workflows, including testing, deployment, and monitoring.
- Perform migration to production deployment from lower environments, test & validate
Must Have Skills
- Must be certified in any ETL tools, Database, Cloud.(Snowflake certified is more preferred)
- Must have implemented at least 3 end-to-end projects in Data Engineering.
- Must have worked on performance management optimization and tuning for data loads, data processes, data transformation in big data
- Must be flexible to write code using JAVA/Scala/Python etc. as required
- Must have implemented CI/CD pipelines using tools like Jenkins, GitLab CI, or AWS CodePipeline.
- Must have managed a team technically of min 5 members and guided the team technically.
- Must have the Technical Ownership capability of Data Engineering delivery.
- Strong communication capabilities with client facing.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- 5 years of experience in software engineering or a related role, with a strong focus on Any ETL Tool, database, integration.
- Proficiency in Any ETL tools like Talend , Informatica etc for Data Integration for building and orchestrating data pipelines.
- Hands-on experience with relational databases such as MySQL, PostgreSQL, or Oracle, and NoSQL databases such as MongoDB, Cassandra, or Redis.
- Solid understanding of database design principles, data modeling, and SQL query optimization.
- Experience with data warehousing, Data Lake , Delta Lake concepts and technologies, data modeling, and relational databases.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Production, Manufacturing & Engineering
Role Category: Engineering
Role: Engineering - Other
Employement Type: Full time
Contact Details:
Company: Edgematics ConsultingLocation(s): Pune
Keyskills: Data Engineering Snowflake ETL Azure Aws Data Management Big Data Ci/Cd Data Integration Data Quality Data Pipeline Data Warehousing Data Modeling Data Governance
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in