Senior Data Engineer
- NCSI Technologies
- 8 - 12 years
- Pune
- 3 months ago
- Email to a friend
- Report this job
Job Description
Essential for this role:Education and Qualifications:
Bachelors degree in IT, Computer Science, Software Engineering, Business Analytics or equivalent.
Minimum seven plus years of experience in data analytics field
Experience with Azure/AWS DatabricksExperience in building and optimizing data pipelines, architectures and data sets
Excellent experience in Scala or Python, PySpark and SQL
Ability to troubleshoot and optimize complex queries on the Spark platform
Knowledgeable on structured and unstructured data design / modelling, data access and data storage techniques
Expertise in designing and deploying data applications on cloud solutions, such as Azure or AWS
Hands on experience in performance tuning and optimizing code running in Databricks environment
Demonstrated analytical and problem-solving skills particularly those that apply to a big data environment Technical / Professional Skills : Excellent communication skills / should interact directly with the customer
Azure/AWS Databricks
Python / Scala / Spark / PySpark
HIVE / HBase / Impala / Parquet
Sqoop, Kafka, Flume
Very Strong SQL and RDBMS
Airflow
Jenkins / Bamboo
Github / Bitbucket
Nexus
Knowledge on Delta Live Tables (DLT) will be an added advantage Good to have:
Relevant accredited certifications for Azure, AWS, Cloud Engineering, and/or Databricks
Data Engineering expertise in Big Data Management using Cloudera, Python, Hive, Scala, Data Warehouse, Data Lake, AWS, Azure
Roles and Responsibilities
1 Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks
2 Ensure data pipelines run smoothly and efficiently
3 Developing scalable and re-usable frameworks for ingesting of data sets
4 Integrating the end-to-end data pipeline - to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times
5 Working with event based / streaming technologies to ingest and process data
6 Working with other members of the project team to support delivery of additional project components (API interfaces, Search)
7 Evaluating the performance and applicability of multiple tools against customer requirements
8 Technically competent in Cloud and Databricks to provide technical advice to the team and to be involved in issue resolution
9 Provide on-call support and afterhours/weekend support as needed
10 Fulfill Service Requests related to Data Analytics platform
11 Lead and drive optimization and continuous improvement initiatives
12 Play gate-keeping role and conduct technical review of the changes as part of release management
13 Understand various data security standards and adhere to the required data security controls in the platform
14 Lead the design, development, and deployment of advanced data pipelines and analytical workflows on the Databricks Lakehouse platform.
15 Collaborate with data scientists, engineers, and business stakeholders to build and scale end-to-end data solutions.
16 Own architectural decisions and ensure adherence to data governance, security, and compliance requirements.
17 Mentor a team of data engineers, providing technical guidance and career development.
Implement CI/CD practices for data engineering pipelines using tools like Azure DevOps, GitHub Actions, or Jenkins.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: NCSI TechnologiesLocation(s): Pune
Keyskills: big data administration hive cloudera analytical scala pyspark data warehousing bitbucket sql cloud applications spark design flume jenkins hbase data lake communication skills python github rdbms data analytics microsoft azure impala data bricks bamboo field kafka sqoop aws
Similar positions
NCSI Technologies
NCS India is a captive unit of NCS Singapore, a member of Singtel Group & Optus. NCS India office is situated in Pune, Maharashtra in India. We provide solutions across a spread of industries including Education, Transportation & Logistics, Shipping, Airport & Aviation, Health...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in