Senior Data Engineer
- Talentship pvt ltd
- 6 - 11 years
- Chennai
- 4 months ago
- Email to a friend
- Report this job
Job Description
Position Overview
We are seeking a highly skilled Senior Data Engineer to join our data platform team, responsible for designing and implementing scalable, cloud-native data solutions. This role focuses on building modern data infrastructure using AWS and Azure services to support our growing analytics and machine learning initiatives.
Key Responsibilities
- Business Translation: Collaborate with business stakeholders to understand requirements and translate them into scalable data models, architectures, and robust end-to-end data pipelines
- ETL/ELT Implementation: Design, develop, and maintain high-performance batch and real-time data pipelines using modern frameworks like Apache Spark, Delta Lake, and streaming technologies
- Data Platform Management: Architect and implement data lakes, data warehouses, and Lakehouse architectures following medallion architecture principles (Bronze, Silver, Gold layers)
Data Operations & Quality
- Pipeline Orchestration: Implement complex workflow orchestration using tools like AWS Step Functions or Azure Data Factory
- Data Quality Assurance: Establish comprehensive data validation, monitoring, and quality frameworks using tools like Great Expectations or custom Python solutions
- Performance Optimization: Monitor, troubleshoot, and optimize data pipeline performance, implementing partitioning strategies and query optimization techniques
- Data Governance: Implement data lineage tracking, metadata management, and ensure compliance with data privacy regulations (GDPR)
Cloud Infrastructure & DevOps
Experience & Background
- 6+ years of hands-on experience in Data Engineering roles with demonstrated expertise in cloud-native data solutions
- Cloud Expertise: Strong preference for AWS/Azure data services
- Production Experience: Proven track record of building and maintaining production-grade data pipelines processing terabytes of data
Technical Skills
Programming & Development
- Python: Advanced proficiency in Python for data engineering (pandas, numpy, boto3, azure-sdk)
- SQL: Expert-level SQL skills including complex queries, window functions, CTEs, and performance tuning
- PySpark: Hands-on experience with PySpark for large-scale data processing and optimization
- Version Control: Proficient with Git workflows, branching strategies, and collaborative development
AWS Data Services (Primary Focus)
- Compute: Experience working on services like AWS Glue, Lambda Azure Data Factory, Synapse Analytics, Data Lake Storage Gen2, S3 Redshift, RDS, DynamoDB
- Analytics: Power BI, Athena, Kinesis (Data Streams, Data Firehose, Analytics)
- Security: IAM, KMS, VPC, Security Groups for secure data access patterns
Soft Skills & Collaboration
- Problem-Solving: Strong analytical and troubleshooting skills with attention to detail
- Communication: Excellent verbal and written communication skills for cross-functional collaboration
- Project Management: Ability to manage multiple priorities and deliver projects on time
- Mentorship: Experience mentoring junior engineers and promoting best practices
Preferred Qualifications (Nice to Have)
- CI/CD Pipelines: Experience with GitHub Actions, Jenkins, or Azure DevOps for automated testing and deployment
- Infrastructure as Code: Terraform or AWS CDK for infrastructure automation
- Containerization: Docker and Kubernetes for containerized data applications
- Testing Frameworks: Unit testing, integration testing, and data quality testing practices
- Cloud Certifications: AWS Certified Data Analytics, Azure Data Engineer Associate, or similar certifications
- Agile Methodologies: Experience working in Scrum/Kanban environments
What We Offer
- Opportunity to work with cutting-edge data technologies and modern cloud platforms
- Collaborative environment with opportunities for professional growth and learning
- Flexible work arrangements and comprehensive benefits package
- Conference attendance and certification reimbursement programs
This role requires the ability to work independently while collaborating effectively with cross-functional teams including Data Scientists, Analytics Engineers, Software Engineers, and Business Stakeholders.
Bottom of Form
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Data Science & AnalyticsRole Category: Data Science & Machine LearningRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Talentship pvt ltdLocation(s): Chennai
Keyskills: powerbi azure-sdk aws RDS Pyspark Data Lake Storage Gen2 S3 Redshift Synapse Analytics AWS Glue DynamoDB numpy kinesis Lambda Azure Data Factory pandas boto3 athena Python
Similar positions
Talentship pvt ltd
About us: https://talentship.io/ Talentship is backed by a team of highly experienced CTOs from well-known digital companies in Germany, and a team of 100+ top-notch developers. With over 25 years of experience in data, software development and infrastructure, we have developed numerous AI appli...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in