Job Description
1.Strategic Leadership:
oDefine and lead the vision for the Databricks Center of Excellence (CoE), positioning Databricks as a strategic data and AI platform for Apexon and its clients.
oDrive the adoption of Databricks for large-scale data processing, analytics, and AI/ML workloads.
2.Architecture and Implementation:
oDesign and implement data architectures leveraging Databricks, Delta Lake, and Spark for big data processing and analytics.
oDevelop frameworks for ETL pipelines, real-time streaming, and batch processing using Databricks.
oOptimize the use of Databricks features, such as MLFlow, AutoML, and Delta Lake for enterprise solutions.
3.Performance and Optimization:
oLead efforts to optimize Spark jobs, cluster configurations, and data storage for performance and cost efficiency.
oEnsure scalability of Databricks solutions to handle increasing data volumes and compute requirements.
4.Integration with Cloud and Ecosystem Tools:
oIntegrate Databricks with cloud platforms (AWS, Azure, GCP) and tools such as Snowflake, Tableau, or Power BI.
oImplement CI/CD pipelines for Databricks workflows and model deployment.
5.Governance and Security:
oImplement robust data governance, access control, and security policies in Databricks environments.
oEnsure compliance with industry regulations and best practices for data privacy.
6.Thought Leadership and Team Development:
oBuild and manage a team of Databricks professionals, fostering innovation and technical excellence.
oStay updated on Databricks advancements and advocate for their adoption through client presentations, workshops, and internal knowledge-sharing.
Technical Competencies:
- Core Expertise: Proficiency in Spark, Delta Lake, and Databricks Workflows.
- Programming: Advanced skills in Python, Scala, and SQL for data engineering tasks.
- Real-Time Processing: Experience with real-time data integration using Kafka, Event Hub, or Databricks Structured Streaming.
- Cloud Ecosystem: Expertise in deploying Databricks on AWS, Azure, or GCP.
Qualifications:
oBachelor s or Master s degree in Data Engineering, Computer Science, or related field.
o10+ years of experience, with 3+ years focused on Databricks-based data solutions.
oDatabricks certifications such as Databricks Certified Data Engineer Professional.
oExperience with multi-cloud and hybrid Databricks deployments.
Job Classification
Industry: IT Services & Consulting
Functional Area / Department: Engineering - Software & QA
Role Category: Software Development
Role: Data Platform Engineer
Employement Type: Full time
Contact Details:
Company: Apexon
Location(s): Pune
Keyskills:
Computer science
Engineering services
Bfsi
Healthcare
Data processing
Life sciences
Asset management
Analytics
SQL
Python