Data Architect/ Lead (Spark or Flink Streaming)
- Photon
- 9 - 14 years
- Hyderabad
- 3 months ago
- Email to a friend
- Report this job
Job Description
Job Description :
- Expertise in Python Language is MUST.
- SQL (should be able to write complex SQL Queries) is MUST
- Hands on experience in Apache Flink Streaming Or Spark Streaming MUST
- Hands On expertise in Apache Kafka experience is MUST
- Data Lake Development experience.
- Orchestration (Apache Airflow is preferred).
- Spark and Hive: Optimization of Spark/PySpark and Hive apps
- Trino/(AWS Athena) (Good to have)
- Snowflake (good to have).
- Data Quality (good to have).
- File Storage (S3 is good to have)
Responsibilities:
- Bachelor's degree in Computer Science, Engineering, or a related field
- 5 + years experience in data engineering, ELT development, and data modeling.
- Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.
- Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.
- Demonstrated experience in developing custom connectors for data ingestion from various sources.
- Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
- Experience implementing DataOps principles and practices, including data CI/CD pipelines.
- Excellent problem-solving and troubleshooting skills, with a strong attention to detail.
- Knowledge of data streaming and real-time data processing technologies (e.g., Apache Kafka).
- Strong understanding of software development principles and practices, including version control (e.g., Git) and code review processes.
- Experience with Agile development methodologies and working in cross-functional Agile teams.
- Ability to adapt quickly to changing priorities and work effectively in a fast-paced environment.
- Excellent analytical and problem-solving skills, with a keen attention to detail.
- Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QA,
Role Category: DBA / Data warehousing
Role: Data warehouse Architect / Consultant
Employement Type: Full time
Contact Details:
Company: PhotonLocation(s): Hyderabad
+ View Contact
Keyskills: streaming data Spark SQL Spark Streaming
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Photon
Photon is the Largest and Fastest growing provider of Mobile First Omnichannel Experiences. We work with over 35% of the F100's (including award winning presence like Walgreens (Best Retail app of the year at SXSW, Webby winner), JP Morgan Chase (winner of multiple Webby's), Western Union, Mastercar...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in