Inno Protech - Data Engineer - SQL/Python @ Alois Technologies

Job Description

We are looking for a skilled Data Engineer with expertise in Big Data technologies to join our dynamic team. As a Data Engineer, you will be responsible for designing, building, and maintaining scalable and efficient data pipelines and systems to support our data processing and analytics needs. The ideal candidate will have a strong background in distributed systems, data modeling, and programming, with hands-on experience in implementing solutions using Big Data technologies such as Hadoop, Spark, and Kafka. Responsibilities Data Pipeline Development : Design and implement scalable data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. Develop ETL (Extract, Transform, Load) processes to cleanse, aggregate, and enrich data from diverse sources. Optimize data pipelines for performance, reliability, and scalability, considering factors such as data volume, velocity, and. variety. Data Modeling And Storage Design and implement data models to support analytical and reporting requirements. Evaluate and select appropriate storage solutions for different types of data (e.g., relational databases, NoSQL databases, data lakes). Implement data partitioning, indexing, and compression techniques to optimize storage and query performance. Big Data Technologies Utilize Big Data technologies such as Hadoop, Spark, Kafka, and HDFS for distributed data processing and storage. Develop and deploy Spark jobs for batch processing, stream processing, and machine learning tasks. Configure and manage Kafka clusters for real-time data ingestion and event streaming. Data Quality And Governance Implement data quality checks and validation rules to ensure accuracy, completeness, and consistency of data. Establish data governance policies and practices to maintain data integrity and compliance with regulatory requirements. Monitor data pipelines and perform troubleshooting to identify and resolve data quality issues. Collaboration And Documentation Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions. Document technical designs, architecture, and implementation details for data pipelines and systems. Provide guidance and mentorship to junior team members on best practices and techniques for Big Data Bachelor's degree in Computer Science, Engineering, or related field. Advanced degree preferred. Proven experience (X years) as a Data Engineer or Big Data Engineer, with expertise in designing and implementing scalable data solutions. Strong programming skills in languages such as Java, Scala, or Python. Hands-on experience with Big Data technologies including Hadoop, Spark, Kafka, and related ecosystem tools (e.g., Hive, HBase, Sqoop). Proficiency in SQL and experience with relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra). Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform. Knowledge of data warehousing concepts and techniques. Excellent problem-solving skills and attention to detail. Strong communication and collaboration skills, with the ability to work effectively in a team environment. If you are passionate about leveraging Big Data technologies to solve complex data challenges and thrive in a fast-paced, collaborative environment, we encourage you to apply for this exciting opportunity. Join our team and contribute to building innovative data solutions that drive business insights and decision-making! (ref:hirist.tech

Employement Category:

Employement Type: Full time
Industry: Others
Role Category: General / Other Software
Functional Area: Not Applicable
Role/Responsibilies: Inno Protech - Data Engineer - SQL/Python

+ View Contact

Keyskills:   distributed systems data modeling programming Hadoop Spark Kafka ETL data modeling data quality data governance SQL relational databases data warehousing Big Data technologies data pipelines data storage NoSQL databases cloud platforms