Azure Data Engineer - Databricks/Power BI @ Gamut infosystems

Job Description

Job Description MSFT Azure Technology stack Azure data factory (ADF) DBT ( Data build Tool) Databricks Power BI SSAS (SQL Server Analysis Services) SQL Server Database SSIS Responsibilities Design, develop, and deploy data processing pipelines on the Azure Databricks platform using Spark-based technologies (e.g., Spark SQL, Spark Streaming). Collaborate with data engineers and data scientists to understand data requirements and design efficient data models and workflows. Optimize and tune Spark jobs to improve performance, scalability, and reliability of data processing applications. Implement data security and governance best practices to ensure compliance with regulatory requirements and protect sensitive data. Develop and maintain ETL processes for data ingestion, transformation, and loading from various sources (e.g., databases, files, streaming data). Work closely with DevOps teams to automate deployment and monitoring of Azure Databricks workloads. using CI/CD pipelines and monitoring tools. Troubleshoot and debug issues in production environments, providing timely resolution and ensuring high availability of data services. Stay updated on industry trends and advancements in big data technologies, cloud computing, and data analytics. Provide technical guidance and support to other team members, including code reviews and knowledge sharing sessions. Collaborate with cross-functional teams to drive innovation and continuous improvement in data processing capabilities. Requirements Bachelor's degree in Computer Science, Engineering, or related field. Proven experience as a data engineer, software engineer, or similar role, with a focus on big data technologies and cloud platforms. Hands-on experience with Azure Databricks, including building and optimizing Spark-based data processing workflows. Proficiency in Apache Spark, Spark SQL, and related technologies for data manipulation and analysis. Strong programming skills in Python, Scala, or Java, with experience in developing data processing applications. Experience with data modeling, schema design, and optimization techniques for large-scale data sets. Knowledge of cloud computing concepts and experience with Azure services (e.g., Azure Data Lake Storage, Azure SQL Database, Azure Synapse Analytics). Familiarity with data security and governance principles, including encryption, access control, and data masking techniques. Excellent problem-solving skills, with the ability to analyze complex data issues and propose effective solutions. Strong communication and collaboration skills, with the ability to work effectively in a team environment and interact with stakeholders at all levels. Preferred Qualifications Certification in Azure Data Engineer or related Azure certifications. Experience with machine learning and data science techniques for predictive analytics and advanced data processing. Knowledge of containerization technologies such as Docker and Kubernetes for deploying and managing distributed applications. Familiarity with NoSQL databases (e.g., MongoDB, Cassandra) and distributed file systems (e.g., Hadoop HDFS). Experience with streaming data processing frameworks such as Apache Kafka or Azure Event Hubs. Understanding of DevOps practices and experience with version control systems (e.g., Git) and CI/CD pipelines (e.g., Azure DevOps, Jenkins). (ref:hirist.tech

Employement Category:

Employement Type: Full time
Industry: Others
Role Category: General / Other Software
Functional Area: Not Applicable
Role/Responsibilies: Azure Data Engineer - Databricks/Power BI

+ View Contact

Keyskills:   DBT Power BI SSAS SSIS Python Scala Java data modeling data security data governance encryption access control data masking communication collaboration machine learning predictive analytics containerization Docker Kubernetes MongoDB Cassandra distributed file systems Apache Kafka Git Azure DevOps Jenkins Azure data factory Databricks SQL Server Database Spark SQL schema design Azure Data Lake Storage Azure SQL Database Azure Synapse Analytics problemsolving NoSQL databases Hadoop HDFS streaming data processing Azure Event Hubs DevOps practices version control systems CICD pipelines