Principal Software Engineer - Data Transfromation
- Cognite
- 8 - 13 years
- Bengaluru
- 3 months ago
- Email to a friend
- Report this job
Job Description
As a Principal Software Engineer, you'll be a technical leader who shapes platform direction, mentors engineering teams, and tackles the most complex data engineering challenges in industrial technology. This role combines deep technical expertise with strategic thinking and leadership influence.
Strategic Impact Leadership
- Technical Strategy Architecture
- Define platform architecture for next-generation industrial data processing capabilities serving 100M+ daily requests
- Drive technology roadmap decisions that impact multiple engineering teams and product areas
- Lead architectural reviews and establish technical standards across the data platform organization
- Champion innovation initiatives that differentiate Cognites technical capabilities in the industrial data market
- Architect foundational systems that serve as building blocks for multiple product teams and use cases
- Design advanced abstractions and frameworks that accelerate development velocity across the organization
- Own end-to-end platform reliability including SLAs, disaster recovery, and operational excellence
- Drive platform observability strategy with comprehensive metrics, alerting, and distributed tracing
- Lead Apache Spark ecosystem contributions - modify core components, contribute to open-source projects, influence roadmaps
- Architect multi-petabyte streaming systems using Flink, Kafka, and custom processing engines for industrial IoT data
- Design custom query engines and DSLs optimized for time-series analytics and operational data patterns
- Pioneer advanced optimization techniques for columnar storage, query planning, and distributed execution
- Mentor Staff and Senior engineers across multiple teams, developing technical leadership capabilities
- Drive cross-functional initiatives involving ML platform, product engineering, and infrastructure teams
- Establish engineering culture around technical excellence, innovation, and customer obsession
- Represent Cognite externally through conference speaking, open- source leadership, and technical thought leadership
Platform Engineering Excellence
Advanced Technical Leadership
Organizational Impact
Exceptional Candidate Profile
- Technical Leadership Mastery (8+ years)
- Distributed Systems Architecture - Advanced Apache Spark expertise - contributed to Spark core, Catalyst optimizer, or execution engine with merged commits - Large-scale streaming architecture - designed and operated Flink/Kafka systems processing 1M+ events/second with complex statemanagement - Custom execution engine development - built domain-specific query engines or processing frameworks from scratch - Performance engineering excellence - consistently delivered 10x+ performance improvements through algorithmic and architectural innovations
- Platform Engineering Leadership - Multi-tenant platform design - architected platforms serving 50+ engineering teams with sophisticated resource isolation and management - Advanced data lake architecture -designed exabyte-scale storage systems with intelligent tiering, compaction, and query optimization - Service mesh and infrastructure - implemented sophisticated microservices architectures with advanced networking, security, and observability
- Data Engineering Innovation - OLAP system expertise - deep experience with ClickHouse, Pinot, Druid internals including cluster management and query optimization - Advanced stream processing - implemented complex event processing, windowing, and stateful computations for industrial use cases - Data modeling excellence - designed sophisticated schemas for dimensional modeling, event sourcing, and real-time analytics
- Strategic Technical Decision Making - Technology evaluation leadership - led organization-wide technology adoption decisions with comprehensive POCs and risk analysis - Architecture governance - established and maintained architectural principles, standards, and review processes - Technical debt management - strategically balanced feature velocity, system reliability, and long-term maintainability
- Open-source contributions to major Apache projects in the data space (e.g. Apache Spark or Kafka) is a big plus - Significant open source contributions - maintainer or core contributor to major Apache projects (Spark, Flink, Kafka, Airflow) - Technical thought leadership - regular speaker at major conferences (Strata, Spark Summit, QCon) with recognized expertise - Community building -organized engineering communities, contributed to standards bodies, or led technical advisory roles Cross-Functional Collaboration - Product partnership excellence - translated ambiguous business requirements into robust technical architectures - Engineering leadership - mentored and developed multiple Senior and Staff engineers with measurable career progression - Executive communication - effectively communicated complex technical concepts to C-level stakeholders and board members
- High-Impact Delivery - Rapid scaling experience - led technical initiatives during hypergrowth phases (10x team/infrastructure scaling) - Customer- facing technical leadership - directly engaged with enterprise customers on complex technical integrations - Business impact ownership - drove technical initiatives that resulted in measurable business outcomes (cost reduction, revenue enablement)
- Innovation Ambiguity Navigation - Greenfield platform development architected major platform components from concept to production at scale - Technical risk management - successfully navigated high-risk technical decisions with incomplete information - Competitive differentiation -developed technical capabilities that created sustainable competitive advantages
Technical Leadership Excellence
Open Source Community Leadership -
Startup Excellence Scale
Technical Environment Challenges
- Advanced Technical Stack
- Core Languages: Kotlin, Scala, Java (advanced JVM optimization)
- Big Data Ecosystem: Apache Spark (internals), Apache Flink, Apache Kafka, Apache Airflow
- Advanced Storage: ClickHouse, PostgreSQL, Elasticsearch, S3-compatible systems with custom optimization
- Infrastructure: Kubernetes (operator development), Terraform,Advanced observability stack
- Table Format Innovation: Apache Iceberg, Delta Lake internals andoptimization
- Query Engine Development: Trino/Presto, Apache Pinot, custom engine development
- Advanced Streaming: Complex event processing, exactly-once semantics, advanced windowing
- ML/AI Integration: Feature stores, model serving infrastructure, MLOps platform integration
- Industrial IoT at Scale: Processing sensor data from millions of industrial assets in real-time
- Complex Time-Series Analytics: Advanced temporal queries, anomaly detection, predictive analytics
- Multi-Cloud Optimization: Cost and performance optimization across AWS, Azure, GCP
- Enterprise Integration: Secure, scalable integration with legacy industrial systems
Cutting-Edge Technologies you'll Drive
Unique Technical Challenges
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Technical ArchitectEmployement Type: Full timeContact Details:
Company: CogniteLocation(s): Bengaluru+ View Contact
Keyskills: Product engineering Data management GCP Pharma Cost reduction Risk management Open source Apache microsoft Distribution system
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in