Big Data Developer
- TechStar Group
- 7 - 10 years
- Hyderabad
- 2 months ago
- Email to a friend
- Report this job
Job Description
Responsibilities of the Candidate :- Be responsible for the design and development of big data solutions. Partner with domain experts, product managers, analysts, and data scientists to develop Big Data pipelines in Hadoop- Be responsible for moving all legacy workloads to a cloud platform- Work with data scientists to build Client pipelines using heterogeneous sources and provide engineering services for data PySpark science applications- Ensure automation through CI/CD across platforms both in cloud and on-premises- Define needs around maintainability, testability, performance, security, quality, and usability for the data platform- Drive implementation, consistent patterns, reusable components, and coding standards for data engineering processes- Convert SAS-based pipelines into languages like PySpark, and Scala to execute on Hadoop and non-Hadoop ecosystems- Tune Big data applications on Hadoop and non-Hadoop platforms for optimal performance- Apply an in-depth understanding of how data analytics collectively integrate within the sub-function as well as coordinate and contribute to the objectives of the entire function.- Produce a detailed analysis of issues where the best course of action is not evident from the information available, but actions must be recommended/taken.- Assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients, and assets, by driving compliance with applicable laws, rules, and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct, and business practices, and escalating, managing and reporting control issues with transparencyRequirements :- 6+ years of total IT experience- 3+ years of experience with Hadoop (Cloudera)/big data technologies- Knowledge of the Hadoop ecosystem and Big Data technologies Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr) - Experience in designing and developing Data Pipelines for Data Ingestion or Transformation using Java Scala or Python.- Experience with Spark programming (Pyspark, Scala, or Java)- Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required.- Proficient in programming in Java or Python with prior Apache Beam/Spark experience a plus. - Hand on experience in CI/CD, Scheduling and Scripting- Ensure automation through CI/CD across platforms both in cloud and on-premises- System level understanding - Data structures, algorithms, distributed storage & compute- Can-do attitude on solving complex business problems, good interpersonal and teamwork skills
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QA
Role Category: Software Development
Role: Data Engineer
Employement Type: Full time
Contact Details:
Company: Techstar GroupLocation(s): Hyderabad
Keyskills: Big Data Hive Data Engineering Data Pipeline PySpark Hadoop Kafka HDFS Spark Python
Similar positions
TechStar Group
Techstar group TechStar Group is a niche Product Development, Business Solutions, and global IT Services firm based in Dallas, TX. At TechStar, we help drive Customer Success by leveraging our expertise in IT Solutions and Business Process improvements.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in