Cloud Data Engineer- AWS-Big Data
- InfoCepts
- 5 - 8 years
- India
- 8 months ago
- Email to a friend
- Report this job
Job Description
Position: Cloud Data Engineer- AWS- Big Data
Purpose of the Position:
You will be a critical member of the Infocepts Cloud Data Architect Team. This position requires a sound knowledge on Big Data using Cloud technologies, EMR, Athena, PySpark, S3, AWS Lambda etc. Strong foundation on database concepts and SQL is also required.
Location: Nagpur/Pune/Chennai/Bangalore
Type of Employment: Full-time
Key Result Areas and Activities:
Technology Assessment and Design:
- Study existing technology landscape and understand current data integration frameworks and do impact assessment for the requirements.
- Should be able to design complex Big Data use cases using AWS Services under guidance of Architect.
- Should be able to assist the architect in reasoning about various architecture choices in terms of cost, performance & durability.
- Should be able to suggest optimizations in existing designs.
- Ensure optimal balance between cost and performance.
Documentation and Stakeholder Communication:
- Project documentation, adheres to quality guidelines & schedules.
- Works hand in hand with Architect & PM for successful delivery of project and provide estimation, scoping, scheduling assistance.
- Articulate design decisions very clearly with stakeholders.
- Perform Proof-of-Concepts and document all observations before proposing a new solution.
- Conduct design review sessions with other teams and identify scope of improvement.
Process Improvement and Automation:
- Suggests automation to improve existing processes.
- Assist junior Data Engineers by providing expert advice or troubleshooting steps whenever required.
- Keep suggesting new ways to improve team productivity.
Training and Knowledge Sharing:
- Create technology-focused training plans whenever required.
- Deliver technology-focused training sessions with team members whenever required.
- Conduct Expert Knowledge Sharing sessions with Client Stakeholders whenever required.
- Assist in designing case study documents whenever required.
Roles and Responsibilities
Essential Skills:
- In-depth knowledge of the following AWS services: S3, EC2, EMR, Athena, AWS Glue, Lambda
- Experience with at least one MPP database: AWS Redshift, Snowflake, SingleStore
- Proficiency in Big Data technologies: Apache Spark, Databricks
- Must have strong programming skills in Python
- Responsible for building data pipelines in AWS And Databricks
- Experience with Big Data table formats, such as Delta Lake (open source)
- Must have very strong SQL skills
- Experience with orchestration tools like Apache Airflow
- Expertise in developing ETL workflows with complex transformations such as SCD, deduplications, aggregations, etc.
- Should be a quick and self-learner, ready to adapt to new AWS services or Big Data technologies as required
- Strong understanding of data warehousing concepts
Desirable Skills:
- Cloud Databases: Snowflake, AWS Aurora
- Big Data: Hadoop, Hive
- Cloud Databases: AWS Aurora
- Associate Level or Professional Level AWS Certification
Qualifications:
- Overall, 5+ years of IT experience
- 3+ years of relevant experience in AWS related project
- Bachelor's degree in computer science, engineering, or related field (Master's degree is a plus)
- Demonstrated continued learning through one or more technical certifications or related methods
Qualities:
- Should have the capability to deep dive and research in various technical related fields
- Self-motivated and focused on delivering outcomes for a fast growing team and firm
- Able to communicate persuasively through speaking, writing, and client presentations
- Able to work in a self-organized and cross-functional team
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: InfoCeptsLocation(s): India
Keyskills: hive aws iam amazon redshift big data technologies pyspark data warehousing emr sql cloud spark hadoop big data etl programming snowflake development python workflow aws glue data bricks amazon ec2 s lambda expressions database creation data warehousing concepts athena aws
Similar positions
InfoCepts
Part of the global G4S security conglomerate, this Delhi-based entity has been operating since 1996. It provides business services such as facilities management, staffing for corporate and administrative roles, and security solutions. It reported revenue of around 31.4 crore in FY 2022 and functions...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in