Data Engineer || Carnation Infotech || Remote
- Carnation Infotech
- 3 - 6 years
- Lucknow
- 3 months ago
- Email to a friend
- Report this job
Job Description
Job Summary
As a Data Engineer , you will be responsible for designing, building, and maintaining the infrastructure and tools that enable the organization to collect, store, and analyze large volumes of data. You will work closely with data scientists, analysts, and other engineers to ensure that our data systems are reliable, efficient, and scalable.
Key Responsibilities:
Data Pipeline Development: Design, develop, and maintain scalable and reliable data pipelines to process large datasets from various sources.
Data Modeling: Collaborate with data scientists and analysts to design and implement data models that support their analytical needs.
ETL Processes: Build and optimize Extract, Transform, Load (ETL) processes to ensure data is accurately ingested and transformed.
Data Quality: Implement data quality checks and validation processes to ensure the integrity of the data.
Data Integration: Integrate data from multiple sources, including APIs, flat files, and databases, into a unified data environment.
Performance Tuning: Monitor and optimize data processing performance to meet the demands of the organization.
Documentation: Create and maintain documentation for data pipelines, data models, and other related processes.
Collaboration: Work closely with cross-functional teams to understand data needs and provide solutions that support business goals.
Compliance and Security: Ensure data handling practices comply with relevant data privacy regulations and security standards.
Qualifications:
Experience
3+ years of experience in data engineering domain.
Technical Skills:
- Experience with ETL process (data extraction, data cleansing, data transformation, data monitoring, data modelling)
- Experience with Microsoft Azure (Azure Data Factory, Azure Databricks, Azure Storage, Azure SQL Database, Azure Synapse)
- Familiarity with Big Data systems (batch & stream processing, Spark, Hive)
- Experience with two or more coding languages (including Python with pyspark)
- Experience with SQL (Spark SQL, T-SQL),
- Familiarity with code versioning (GIT, Azure DevOps)
- Experience with various data formats (e.g. CSV, JSON, Parquet, Delta Lake)
- Knowledge of distributed systems concepts and principles (consistency, availability, safety, durability, reliability, fault-tolerance)
Soft Skills:
- Strong problem-solving and analytical skills
- Excellent communication and collaboration skills
- Ability to work in a fast-paced, dynamic environment
Eager to learn new things and passionate about technology
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Carnation InfotechLocation(s): Lucknow
Keyskills: Azure Data Factory Pyspark Azure Databricks SQL ETL
Similar positions
Carnation Infotech
Carnation Infotech a sister company of a New Jersey based IT and Staffing consulting services, Cardinal Technology Inc services to Fortune 1000 organizations. We service large corporate, Fortune 1000 companies such as Johnson & Johnson, Opko Health Inc., AT&T, NBC, Bed Bath & Beyond, etc...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in