Principal Software Engineer - Data Transfromation

6 Minutes ago โ€ข 8 Years +
Data Analysis

Job Description

Cognite is a global SaaS leader leveraging AI and data through Cognite Atlas AI and Cognite Data Fusion (CDF) to solve complex industrial challenges. Awarded for technology innovation, Cognite is at the forefront of digital transformation in sectors like Oil & Gas and Manufacturing. This role seeks a Principal Software Engineer to drive technical strategy, architectural excellence, and cross-organizational impact, shaping the platform direction and mentoring teams while tackling complex data engineering challenges in industrial technology.
Good To Have:
  • Open-source contributions to major Apache projects in the data space (e.g. Apache Spark or Kafka).
  • Regular speaker at major conferences (Strata, Spark Summit, QCon).
  • Organized engineering communities, contributed to standards bodies, or led technical advisory roles.
Must Have:
  • Define platform architecture for next-generation industrial data processing capabilities.
  • Drive technology roadmap decisions impacting multiple engineering teams and product areas.
  • Lead architectural reviews and establish technical standards across the data platform organization.
  • Architect foundational systems for multiple product teams and use cases.
  • Design advanced abstractions and frameworks to accelerate development velocity.
  • Own end-to-end platform reliability including SLAs, disaster recovery, and operational excellence.
  • Drive platform observability strategy with comprehensive metrics, alerting, and distributed tracing.
  • Lead Apache Spark ecosystem contributions.
  • Architect multi-petabyte streaming systems using Flink, Kafka, and custom processing engines.
  • Design custom query engines and DSLs optimized for time-series analytics.
  • Pioneer advanced optimization techniques for columnar storage, query planning, and distributed execution.
  • Mentor Staff and Senior engineers across multiple teams.
  • Drive cross-functional initiatives involving ML platform, product engineering, and infrastructure teams.
  • Establish engineering culture around technical excellence, innovation, and customer obsession.
  • 8+ years of Technical Leadership Mastery.
  • Expertise in Distributed Systems Architecture, Large-scale streaming architecture, Custom execution engine development, Performance engineering.
  • Experience in Multi-tenant platform design, Advanced data lake architecture, Service mesh and infrastructure.
  • Expertise in OLAP system, Advanced stream processing, Data modeling.
  • Strategic Technical Decision Making, Architecture governance, Technical debt management.
  • High-Impact Delivery, Rapid scaling experience, Customer-facing technical leadership, Business impact ownership.
  • Innovation & Ambiguity Navigation, Greenfield platform development, Technical risk management, Competitive differentiation.
Perks:
  • Join an organization of 70 different nationalities with Diversity, Equality and Inclusion (DEI) in focus.
  • Office location Rathi Legacy (Rohan Tech Park) Hoodi (Bengaluru).
  • A highly modern and fun working environment with sublime culture across the organization.
  • Flat structure with direct access to decision-makers, with minimal amount of bureaucracy.
  • Opportunity to work with and learn from some of the best people on some of the most ambitious projects found anywhere, across industries.
  • Join our HUB to be part of the conversation directly with Cogniters and our partners.
  • Hybrid work environment globally.

Add these skills to join the top 1% applicants for this job

saas-business-models
team-management
cross-functional
data-analytics
risk-management
game-texts
cross-functional-collaboration
postgresql
networking
apache-flink
aws
service-mesh
azure
model-serving
apache-kafka
terraform
elasticsearch
spark
microservices
kubernetes
kotlin
scala
java

About Cognite

Embark on a transformative journey with Cognite, a global SaaS forerunner in leveraging AI and data to unravel complex business challenges through our cutting-edge offerings including Cognite Atlas AI, an industrial agent workbench, and the Cognite Data Fusion (CDF) platform. We were awarded the 2022 Technology Innovation Leader for Global Digital Industrial Platforms & Cognite was recognized as 2024 Microsoft Energy and Resources Partner of the Year. In the realm of industrial digital transformation, we stand at the forefront, reshaping the future of Oil & Gas, Chemicals, Pharma and other Manufacturing and Energy sectors. Join us in this venture where AI and data meet ingenuity, and together, we forge the path to a smarter, more connected industrial future.

Learn more about Cognite here

Our values

Impact: Cogniters strive to make an impact in all that they do. We are result-oriented, always asking ourselves.

Ownership: Cogniters embrace a culture of ownership. We go beyond our comfort zones to contribute to the greater good, fostering inclusivity and sharing responsibilities for challenges and success.

Relentless: Cogniters are relentless in their pursuit of innovation. We are determined and deliverable (never ruthless or reckless), facing challenges head-on and viewing setbacks as opportunities for growth.

About Cognite & This Role

Cognite is revolutionizing industrial data management through our flagship product, Cognite Data Fusion - a state-of-the-art SaaS platform that transforms how industrial companies leverage their data. We're seeking an exceptional Staff Data Platform Engineer who will drive technical strategy, architectural excellence, and cross-organizational impact.

As a Principal Software Engineer, you'll be a technical leader who shapes platform direction, mentors engineering teams, and tackles the most complex data engineering challenges in industrial technology. This role combines deep technical expertise with strategic thinking and leadership influence.

Strategic Impact & Leadership

  • Technical Strategy & Architecture
  • Define platform architecture for next-generation industrial data processing capabilities serving 100M+ daily requests
  • Drive technology roadmap decisions that impact multiple engineering teams and product areas
  • Lead architectural reviews and establish technical standards across the data platform organization
  • Champion innovation initiatives that differentiate Cognite's technical capabilities in the industrial data market
  • Platform Engineering Excellence
  • Architect foundational systems that serve as building blocks for multiple product teams and use cases
  • Design advanced abstractions and frameworks that accelerate development velocity across the organization
  • Own end-to-end platform reliability including SLAs, disaster recovery, and operational excellence
  • Drive platform observability strategy with comprehensive metrics, alerting, and distributed tracing
  • Advanced Technical Leadership
  • Lead Apache Spark ecosystem contributions - modify core components, contribute to open-source projects, influence roadmaps
  • Architect multi-petabyte streaming systems using Flink, Kafka, and custom processing engines for industrial IoT data
  • Design custom query engines and DSLs optimized for time-series analytics and operational data patterns
  • Pioneer advanced optimization techniques for columnar storage, query planning, and distributed execution
  • Organizational Impact
  • Mentor Staff and Senior engineers across multiple teams, developing technical leadership capabilities
  • Drive cross-functional initiatives involving ML platform, product engineering, and infrastructure teams
  • Establish engineering culture around technical excellence, innovation, and customer obsession
  • Represent Cognite externally through conference speaking, open- source leadership, and technical thought leadership

Exceptional Candidate Profile

  • Technical Leadership Mastery (8+ years)
  • Distributed Systems Architecture - Advanced Apache Spark expertise - contributed to Spark core, Catalyst optimizer, or execution engine with merged commits - Large-scale streaming architecture - designed and operated Flink/Kafka systems processing 1M+ events/second with complex statemanagement - Custom execution engine development - built domain-specific query engines or processing frameworks from scratch - Performance engineering excellence - consistently delivered 10x+ performance improvements through algorithmic and architectural innovations
  • Platform Engineering Leadership - Multi-tenant platform design - architected platforms serving 50+ engineering teams with sophisticated resource isolation and management - Advanced data lake architecture -designed exabyte-scale storage systems with intelligent tiering, compaction, and query optimization - Service mesh and infrastructure - implemented sophisticated microservices architectures with advanced networking, security, and observability
  • Data Engineering Innovation - OLAP system expertise - deep experience with ClickHouse, Pinot, Druid internals including cluster management and query optimization - Advanced stream processing - implemented complex event processing, windowing, and stateful computations for industrial use cases - Data modeling excellence - designed sophisticated schemas for dimensional modeling, event sourcing, and real-time analytics

Technical Leadership Excellence

  • Strategic Technical Decision Making - Technology evaluation leadership - led organization-wide technology adoption decisions with comprehensive POCs and risk analysis - Architecture governance - established and maintained architectural principles, standards, and review processes - Technical debt management - strategically balanced feature velocity, system reliability, and long-term maintainability

Open Source & Community Leadership

  • Open-source contributions to major Apache projects in the data space (e.g. Apache Spark or Kafka) is a big plus - Significant open source contributions - maintainer or core contributor to major Apache projects (Spark, Flink, Kafka, Airflow) - Technical thought leadership - regular speaker at major conferences (Strata, Spark Summit, QCon) with recognized expertise - Community building -organized engineering communities, contributed to standards bodies, or led technical advisory roles Cross-Functional Collaboration - Product partnership excellence - translated ambiguous business requirements into robust technical architectures - Engineering leadership - mentored and developed multiple Senior and Staff engineers with measurable career progression - Executive communication - effectively communicated complex technical concepts to C-level stakeholders and board members

Startup Excellence & Scale

  • High-Impact Delivery - Rapid scaling experience - led technical initiatives during hypergrowth phases (10x team/infrastructure scaling) - Customer- facing technical leadership - directly engaged with enterprise customers on complex technical integrations - Business impact ownership - drove technical initiatives that resulted in measurable business outcomes (cost reduction, revenue enablement)
  • Innovation & Ambiguity Navigation - Greenfield platform development architected major platform components from concept to production at scale - Technical risk management - successfully navigated high-risk technical decisions with incomplete information - Competitive differentiation -developed technical capabilities that created sustainable competitive advantages

Technical Environment & Challenges

  • Advanced Technical Stack
  • Core Languages: Kotlin, Scala, Java (advanced JVM optimization)
  • Big Data Ecosystem: Apache Spark (internals), Apache Flink, Apache Kafka, Apache Airflow
  • Advanced Storage: ClickHouse, PostgreSQL, Elasticsearch, S3-compatible systems with custom optimization
  • Infrastructure: Kubernetes (operator development), Terraform,Advanced observability stack
  • Cutting-Edge Technologies You'll Drive
  • Table Format Innovation: Apache Iceberg, Delta Lake internals andoptimization
  • Query Engine Development: Trino/Presto, Apache Pinot, custom engine development
  • Advanced Streaming: Complex event processing, exactly-once semantics, advanced windowing
  • ML/AI Integration: Feature stores, model serving infrastructure, MLOps platform integration
  • Unique Technical Challenges
  • Industrial IoT at Scale: Processing sensor data from millions of industrial assets in real-time
  • Complex Time-Series Analytics: Advanced temporal queries, anomaly detection, predictive analytics
  • Multi-Cloud Optimization: Cost and performance optimization across AWS, Azure, GCP
  • Enterprise Integration: Secure, scalable integration with legacy industrial systems

Join the global Cognite community! ๐ŸŒ

  • Join an organization of 70 different nationalities ๐ŸŒ with Diversity, Equality and Inclusion (DEI) in focus ๐Ÿค
  • Office location Rathi Legacy (Rohan Tech Park ) Hoodi (Bengaluru)
  • A highly modern and fun working environment with sublime culture across the organization, follow us on Instagram @cognitedata ๐Ÿ“ท to know more
  • Flat structure with direct access to decision-makers, with minimal amount of bureaucracy
  • Opportunity to work with and learn from some of the best people on some of the most ambitious projects found anywhere, across industries
  • Join our HUB ๐Ÿ—ฃ๏ธ to be part of the conversation directly with Cogniters and our partners.
  • Hybrid work environment globally

Why choose Cognite? ๐Ÿ† ๐Ÿš€

Join us in making a real and lasting impact in one of the most exciting and fastest-growing new software companies in the world. We have repeatedly demonstrated that digital transformation, when anchored on strong DataOps, drives business value and sustainability for clients and allows front-line workers, as well as domain experts, to make better decisions every single day. We were recognized as one of CNBC's top global enterprise technology startups powering digital transformation ! And just recently, Frost & Sullivan named Cognite a Technology Innovation Leader ! ๐Ÿฅ‡ Most recently Cognite Data Fusionยฎ Achieved Industry First DNV Compliance for Digital Twins ๐Ÿฅ‡

Apply today!

If you're excited about the opportunity to work at Cognite and make a difference in the tech industry, we encourage you to apply today! We welcome candidates of all backgrounds and identities to join our team.

We encourage you to follow us on Cognite LinkedIn ; we post all our openings there.

Set alerts for more jobs like Principal Software Engineer - Data Transfromation
Set alerts for new jobs by Cognite
Set alerts for new Data Analysis jobs in India
Set alerts for new jobs in India
Set alerts for Data Analysis (Remote) jobs

Contact Us
hello@outscal.com
Made in INDIA ๐Ÿ’›๐Ÿ’™