Senior Big Data Developer - Youappi
- Affle
- 7 - 10 years
- Noida, Gurugram
- 8 months ago
- Email to a friend
- Report this job
Job Description
As a key member of the Big Data team, you will be responsible for architecting and optimizing distributed systems, leveraging modern cloud-native technologies, and ensuring high availability and fault tolerance in our data infrastructure.
Primary Responsibilities:
- Design, develop, and maintain real-time and batch processing pipelines using Apache Spark, Kafka, and Kubernetes.
- Architect high-throughput distributed systems that handle large-scale data ingestion and processing.
- Work extensively with AWS services, including Kinesis, DynamoDB, ECS, S3, and Lambda.
- Manage and optimize containerized workloads using Kubernetes (EKS) and ECS.
- Implement Kafka-based event-driven architectures to support scalable, low-latency applications.
- Ensure high availability, fault tolerance, and resilience of data pipelines.
- Work with MySQL, Elasticsearch, Aerospike, Redis, and DynamoDB to store and retrieve massive datasets efficiently.
- Automate infrastructure provisioning and deployment using Terraform, Helm, or CloudFormation.
- Optimize system performance, monitor production issues, and ensure efficient resource utilization.
- Collaborate with data scientists, backend engineers, and DevOps teams to support advanced analytics and machine learning initiatives.
- Continuously improve and modernize the data architecture to support growing business needs.
Required Skills:
- 7-10+ years of experience in big data engineering or distributed systems development.
- Expert-level proficiency in Scala, Java, or Python.
- Deep understanding of Kafka, Spark, and Kubernetes in large-scale environments.
- Strong hands-on experience with AWS (Kinesis, DynamoDB, ECS, S3, etc.).
- Proven experience working with highly loaded, low-latency distributed systems.
- Experience with Kafka, Kinesis, Flink, or other streaming technologies for event-driven architectures.
- Expertise in SQL and database optimizations for MySQL, Elasticsearch, and NoSQL stores.
- Strong experience in automating infrastructure using Terraform, Helm, or CloudFormation.
- Experience managing production-grade Kubernetes clusters (EKS).
- Deep knowledge of performance tuning, caching strategies, and data consistency models.
- Experience working in a startup environment, adapting to rapid changes and building scalable solutions from scratch.
Nice to Have
- Experience with machine learning pipelines and AI-driven analytics.
- Knowledge of workflow orchestration tools such as Apache Airflow.
Job Classification
Industry: Software ProductFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Software Development - OtherEmployement Type: Full timeContact Details:
Company: AffleLocation(s): Noida, Gurugram
Keyskills: kubernetes python aws iam performance tuning dynamo db scala data engineering redis helm sql nosql database optimization elastic search java system lambda expressions ecs spark automating kafka caching techniques mysql aws big data
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in