Sr. Data Engineer- AWS- Big Data
- InfoCepts
- 7 - 9 years
- Kolkata
- 3 months ago
- Email to a friend
- Report this job
Job Description
You will be a critical member of the Infocepts Cloud Data Architect Team. This position requires a sound knowledge on Big Data using Cloud technologies, EMR, Athena, PySpark, S3, AWS Lambda etc. Strong foundation on database concepts and SQL is also required.
Location: Nagpur/Pune/Chennai/Bangalore
Type of Employment: Full-time
Key Result Areas and Activities:
Technology Assessment and Design:
- Study existing technology landscape and understand current data integration frameworks and do impact assessment for the requirements.
- Should be able to design complex Big Data use cases using AWS Services under guidance of Architect.
- Should be able to assist the architect in reasoning about various architecture choices in terms of cost, performance & durability.
- Should be able to suggest optimizations in existing designs.
- Ensure optimal balance between cost and performance.
Documentation and Stakeholder Communication:
- Project documentation, adheres to quality guidelines & schedules.
- Works hand in hand with Architect & PM for successful delivery of project and provide estimation, scoping, scheduling assistance.
- Articulate design decisions very clearly with stakeholders.
- Perform Proof-of-Concepts and document all observations before proposing a new solution.
- Conduct design review sessions with other teams and identify scope of improvement.
Process Improvement and Automation:
- Suggests automation to improve existing processes.
- Assist junior Data Engineers by providing expert advice or troubleshooting steps whenever required.
- Keep suggesting new ways to improve team productivity.
Training and Knowledge Sharing:
- Create technology-focused training plans whenever required.
- Deliver technology-focused training sessions with team members whenever required.
- Conduct Expert Knowledge Sharing sessions with Client Stakeholders whenever required.
- Assist in designing case study documents whenever required.
Essential Skills:
- In-depth knowledge of the following AWS services: S3, EC2, EMR, Athena, AWS Glue, Lambda
- Experience with at least one MPP database: AWS Redshift, Snowflake, SingleStore
- Proficiency in Big Data technologies: Apache Spark, Databricks
- Must have strong programming skills in Python
- Responsible for building data pipelines in AWS And Databricks
- Experience with Big Data table formats, such as Delta Lake (open source)
- Must have very strong SQL skills
- Experience with orchestration tools like Apache Airflow
- Expertise in developing ETL workflows with complex transformations such as SCD, deduplications, aggregations, etc.
- Should be a quick and self-learner, ready to adapt to new AWS services or Big Data technologies as required
- Strong understanding of data warehousing concepts
Desirable Skills:
- Cloud Databases - Snowflake, AWS Aurora
- Big Data - Hadoop, Hive
- Cloud Databases- AWS Aurora
- Associate Level or Professional Level AWS Certification
Qualifications:
- Overall, 7+ years of IT experience
- 5+ years of relevant experience in AWS related project
- Bachelors degree in computer science, engineering, or related field (Masters degree is a plus)
- Demonstrated continued learning through one or more technical certifications or related methods
Qualities:
- Should have the capability to deep dive and research in various technical related fields
- Self-motivated and focused on delivering outcomes for a fast growing team and firm
- Able to communicate persuasively through speaking, writing, and client presentations
- Able to work in a self-organized and cross-functional team
Job Classification
Industry: Management ConsultingFunctional Area / Department: Engineering - Software & QA
Role Category: Software Development
Role: Data Engineer
Employement Type: Full time
Contact Details:
Company: InfoCeptsLocation(s): Kolkata
+ View Contact
Keyskills: Computer science Automation Process improvement Scheduling Troubleshooting Apache Open source big data SQL Python
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
InfoCepts
InfoCepts Technologies Pvt. Ltd. InfoCepts is a leading provider of Information Management and Business Analytics services. With over 700+ Industry leading professionals spread globally, we help organizations make business decisions faster, smarter & better, by deriving maximum value from th...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in