Data Engineer And Architect (aws/azure/gcp)
- Encycdata
- 5 - 10 years
- Bengaluru
- 4 months ago
- Email to a friend
- Report this job
Job Description
Role & responsibilities
- Build and deploy data platforms, data warehouses, and big data solutions
- Integrating, transforming, and consolidating data from various structured and unstructured data systems.
- Expert in data ingestion, transformation, storage, and analysis, often using Azure services and migration from legacy on-premise services
- Essential skills include SQL, Python, R and knowledge of ETL/ELT processes and big data technologies like Apache Spark, Scala, PySpark.
- Maintain data integrity, resolve data-related issues, and ensure the reliability and performance of data solutions
- Work with stakeholders to provide real-time data analytics, monitor data pipelines, and optimize performance and scalability
- Strong understanding of data management fundamentals, data warehousing, and data modeling.
- Bigdata technologies HDFS, Spark, Hbase, Hive, Sqoop, Kafka, RabbitMQ, Flink
- Implement seamless data integration solutions between Azure/AWS/GCP and Snowflake platforms.
- Identify and resolve performance bottlenecks, optimize queries, and ensure the overall efficiency of data pipelines.
- Lead the development and management of data infrastructure, including tools, dashboards, queries, reports, and scripts, ensuring automation of recurring tasks while maintaining data quality and integrity
- Implement and maintain data security measures, ensuring compliance with industry standards and regulations.
- Ensure data architecture aligns with business requirements and best practices.
- Experience in Power BI /Tableau /Looker
- Management, administration, and maintenance with data streaming tools such as Kafka/Confluent Kafka, Flink
- Experience in Test Driven Development, building libraries, and proficiency in Pandas, NumPy, Elasticsearch, Apache Beam
- Familiarity with CI/CD pipelines, monitoring, and infrastructure-as-code (e. g., Terraform, CloudFormation).
Proficient in query optimization, data partitioning, indexing strategies, and caching mechanisms. - Ensure GDPR, SOX, and other regulatory compliance across data workflows
- Essential skills include SQL, Python, and knowledge of ETL/ELT processes and big data technologies like Spark.
Must have Skills (Atleast one or two)
- Azure Data Factory (ADF), Databricks, Synapse, Data Lake Storage, Timeseries Insights, Azure SQL Database, SQL Server, Presto, SSIS
- AWS data services such as S3, Glue Studio, Redshift, Athena, and EMR, Redshift , Airflow, IAM, DBT, Lambda, RDS, DynamoDB, Neo4j, Amazon Neptune
- GCP Big query, SQL, Composer, Dataflow, Dataform, DBT, /Python, Cloud functions, Dataproc+pyspark, Python injection, Dataflow, Cloud Storage, Pub/Sub, and Vertex AI, GKE, Cloud Functions
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Data Science & AnalyticsRole Category: Data Science & Machine LearningRole: Data EngineerEmployement Type: Full timeContact Details:
Company: EncycdataLocation(s): Bengaluru
Keyskills: Data Engineering Airflow Power Bi Bigquery Kafka Azure Databricks Tableau Machine Learning Azure Synapse NoSQL Data Pipeline Redshift Aws SCALA Data Flow Azure Data Lake Data Modeling Data Architecture Python Pyspark Hadoop Big Data Dataproc SQL Azure Data Factory Glue ETL
Similar positions
Encycdata
Encycdata is a startup company developing proprietory products for EdTech, MedTech, Finops and Event management. We are specialized in software development, IT conculting and staffing to provide top talent to global clients and partners.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in