Senior/ Lead Data Engineer- Azure Blackstraw- Immediate Joiner
- Blackstraw
- 10 - 15 years
- Pallavaram
- 1 month ago
- Email to a friend
- Report this job
Job Description
Data Engineering Lead- Azure (Data Brick)
Company Name: Blackstraw.ai
Job Type: Full-time
Experience: 10 - 15 Years
Candidates who can join immediately will be preferred.
Job Description:
As a lead data engineer you will oversee data architecture, ETL processes, and analytics
pipelines, ensuring efficiency, scalability, and quality.
Key Responsibilities:
Job Requirements:
- Participate in the customers system design meetings and collect the functional/technical requirements.
- Build up data pipelines for consumption by the data science team.
- Skillful in ETL process and tools.
- Clear understanding and experience with Azure Data Factory, Python and PySpark or Spark, with HIVE, Airflow, Impala, and Hadoop and RDBMS architecture.
- Experience in writing Python programs and SQL queries.
- Experience in SQL Query tuning.
- Experienced in Shell Scripting(Unix/Linux).
- Build and maintain data pipelines in Spark/Pyspark with SQL and Python or SCALA.
- Knowledge of Cloud (Azure/AWS/GCP, etc..) technologies is additional.
- Good to have knowledge of Kubernetes, CI/CD concepts, Apache Kafka
- Suggest and implement best practices in data integration.
- Guide the QA team in defining system integration tests as needed.
- Split the planned deliverables into tasks and assign them to the team.
- Needs to Maintain/Deploy the ETL code and follow the Agile methodology
- Needs to work on optimization wherever applicable.
- Good oral, written and presentation skills.
Preferred Qualifications:
- Degree in Computer Science, IT, or similar field; a Masters is a plus.
- Hands-on experience with Python and Pyspark, Azure Data Factory
- Hands-on experience with Spark
- Great numerical and analytical skills.
- Working knowledge of cloud platforms such as MS Azure, AWS, etc...
- Technical expertise with data models, data mining, and segmentation techniques.
Tech Stack:
Minimum of 5+ years working experience with Spark, Hadoop eco systems.
Minimum of 4+ years working experience on designing data streaming pipelines.
Should be an expert in either Python/Scala/Java.
Should have experience in Data Ingestion and Integration into data lake using hadoop ecosystem
tools such as Sqoop, Spark, SQL, Hive, Airflow, etc..
Should have experience optimizing (Performance tuning) data pipelines.
Should have minimum experience of 3+ years on NoSQL and Spark Streaming.
Knowledge of Kubernetes and Docker is a plus.
Should have experience with Cloud services either Azure/AWS.
Should have experience with on-prem distribution such as Cloudera/HortonWorks/MapR.
Basic understanding of CI/CD pipelines.
Basic knowledge of Linux environment and commands.
Preferred Qualifications:
Bachelors degree in computer science or related field.Key Traits:
Should have excellent communication skills.
Should be self motivated and willing to work as part of a team.
Should be able to collaborate and coordinate with on shore and offshore teams.
Be a problem solver and be proactive to solve the challenges that come his way.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: BlackstrawLocation(s): Pallavaram
Keyskills: Pyspark Azure Databricks Spark Python Azure Data Factory Data Bricks
Similar positions
Blackstraw
\n\n Tech Mahindra is an Indian multinational information technology services and consulting company. Part of the Mahindra Group, the company is headquartered in Pune and has its registered office in Mumbai. Tech Mahindra is a US$6.0 billion company[6][7] with over 158,000 employees across 90 countr...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in