Data Engineer - Big Data Technologies Job in
- Kpr sugar apperals
- 5 to 9 Yrs
- Other Maharashtra
- 10 months ago
- Email to a friend
- Report this job
Job Description
- Job Description We are seeking an experienced Data Engineer to join our team and lead the development of a cutting-edge data platform. The platform will leverage distributed computing frameworks such as Apache Spark, Databricks, and Snowflake to enable near real time supply chain planning, eventually leading to advanced analytics, insights into data with the adoption of Generative AI (GenAI) technologies across our product base. Key Responsibilities As a Lead Data Engineer, the candidate would be responsible for : Design and build a highly scalable, fault-tolerant data platform optimized for distributed computing and large-scale data processing Implement data pipelines and ETL/ELT processes using distributed computing frameworks to efficiently ingest, transform, and load massive datasets from various sources Leverage cloud data platforms to enable seamless data sharing, near-zero maintenance, and fast analytics on structured and semi-structured data. Collaborate with data scientists, machine learning engineers, and software developers to understand data requirements and build solutions to power GenAI applications Optimize distributed computing jobs and queries for maximum performance and cost efficiency Implement data governance, security, and compliance best practices Provide guidance on distributed computing architecture and mentor junior data engineers Qualifications 5+ years of experience as a Data Engineer working with big data technologies. Strong proficiency in SQL, Python programming and data modeling techniques. Deep expertise in distributed computing principles and frameworks (e.g. Apache Spark), including SQL, streaming, and optimizing jobs for scale and efficiency. Hands-on experience with developing and deploying distributed computing applications using cloud-based platforms (e.g. AWS EMR, Azure HDInsight, or equivalent). Strong understanding of cloud data platform architectures and best practices for ELT/ETL, data sharing, and query optimization (e.g. AWS Athena, AWS Glue, Azure Synapse Analytics, or equivalent). Experience enabling application engineers to build applications leveraging the data platform through APIs and abstractions. Experience with orchestration frameworks like Apache Airflow and data streaming technologies like Kafka Experience building and optimizing data pipelines for machine learning applications. Knowledge of data modelling, data warehousing, and schema design. Familiarity with public cloud platforms such as AWS, Azure, or GCP. Excellent problem-solving and communication skills. Bachelor's or Master's degree in Computer Science (Preferred), Engineering, or a related field (ref:hirist.tech),
Employement Category:
Employement Type: Full timeIndustry: IT Services & ConsultingRole Category: Not SpecifiedFunctional Area: Not SpecifiedRole/Responsibilies: Data Engineer - Big Data Technologies Job inContact Details:
Company: Edgematics GroupLocation(s): Other Maharashtra+ View Contact
Keyskills: SQL data modeling distributed computing Apache Spark Kafka data warehousing Python programming schema design
Fraud Alert to job seekers!
₹ Not Specified
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in