AWS Data Integration-Senior Associate @ Florican Enterprises

Job Description

Key Responsibilities - Data Pipelines Construction: - Design and build efficient, reliable, and scalable data pipelines using Apache Airflow. - Develop complex ETL jobs that process and transform large datasets using PySpark and DBT. - AWS Glue ETL Development: - Design, develop, and maintain scalable ETL jobs using AWS Glue. - Utilize AWS Glue Data Catalog for metadata storage and schema versioning. - Databrew Recipe Creation: - Create and manage Databrew recipes for data preparation and cleansing. - Leverage Databrew blueprints to standardize data transformation processes. - Data Quality Framework: - Architect and implement data quality rules using AWS Glue Data Quality and Databrew. - Build automated checks and balances for data validation and anomaly detection. - Data Transformation Expertise: - Leverage PySpark to perform intricate data manipulations and aggregations at scale. - Utilize DBT to manage database transforms declaratively with a focus on modularity and. reusability. - Testing and Maintenance: - Conduct unit tests and develop database validation scripts to ensure data quality and integrity. - Troubleshoot and resolve issues in our data pipelines and propose improvements. - Process Automation: - Automate data processes using Python scripting, focusing on AWS SDK For Pandas for data analysis and manipulation tasks. - Implement best practices in version control, testing, deployment, and orchestration of data pipeline workflow. - Automate manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. - AWS Analytics Integration: - Create and maintain optimal data pipeline architecture for AWS services such as S3, Glue, DataBrew, DQ Rules, etc. - Ensure seamless integration of data transformation tools with AWS analytics components. - Maintain Data Quality: - Establish data quality checks and balance monitoring for all components of pipeline to ensure accurate and available information. - Troubleshoot and remediate data processing issues and performance bottlenecks. - Collaboration and Support: - Work with stakeholders including data, design, product and executive teams to assist with data-related technical issues and support their data infrastructure needs. - Document data pipeline processes and contribute to shared knowledge within the team. Skills & Qualifications: Minimum of 5 years of hands-on experience with AWS Cloud Services, especially related to data storage, processing and analytics. In-depth knowledge of AWS services for Data Lakehouse Constructions; Expertise in AWS services like AWS S3 for data storage, AWS Glue for data cataloging and ETL, AWS Lake Formation for setting up secure data lake with granular access control, and other relevant AWS analytics services. Proficiency in Apache Airflow: Strong experience in using Apache Airflow to create, schedule, and monitor directed acyclic graphs (DAGs) for orchestrating complex data pipelines. This includes tasks such as data ingestion, transformation, and implementation of data quality rulesets. Data Pipelines and Data Product Management: Skilled in designing and building robust data pipelines and data products, ensuring efficient data flow and processing. Ability to implement data transformation with PySpark and data quality rules to maintain high standards of data integrity. Experience with Data Lakehouse Architecture: Proven capability in managing a Data Lakehouse environment on AWS. Certifications in AWS (e.g., AWS Certified Solutions Architect Professional, AWS Certified Data Analytics Specialty) Proven track record of designing and implementing data Lakehouse solutions in a financial context. Soft Skills: 1. Team Player ability to work in global team environment. Partner with Production Support, DevOps teams and business users. 2. Flexible to pick up conflicting priorities 4. Upskill & Upgrade self as per strategic roadmaps 5. Good Communication skills 6. Demonstrate ownership to deliver quality solutions. 7. Open to support during weekends as and when required (only in case of implementation/critical issues

Employement Category:

Employement Type: Full time
Industry: Others
Role Category: Application Programming / Maintenance
Functional Area: Not Applicable
Role/Responsibilies: AWS Data Integration-Senior Associate

+ View Contact

Keyskills:   DBT DBT Python Data Quality Data Transformation Apache Airflow PySpark AWS Glue AWS Glue Data Catalog Databrew PySpark AWS SDK AWS S3 AWS Lake Formation Data Lakehouse Architecture AWS Certified Solutions Architect Professional AWS Certified Data Analytics Specialty