Data Engineer- Lead Data Engineer

Role Overview

We are seeking an experienced Lead Data Engineer to join our Data Engineering team at Paytm, India's leading digital payments and financial services platform. This is a critical role responsible for designing, building, and maintaining large-scale, real-time data streams that process billions of transactions and user interactions daily. Data accuracy and stream reliability are essential to our operations, as data quality issues can result in financial losses and impact customer trust.

As a Lead Data Engineer at Paytm, you will be responsible for building robust data systems that support India's largest digital payments ecosystem. You'll architect and implement reliable, real-time data streaming solutions where precision and data correctness are fundamental requirements. Your work will directly support millions of users across merchant payments, peer-to-peer transfers, bill payments, and financial services, where data accuracy is crucial for maintaining customer confidence and operational excellence.

This role requires expertise in designing fault-tolerant, scalable data architectures that maintain high uptime standards while processing peak transaction loads during festivals and high-traffic events. We place the highest priority on data quality and system reliability, as our customers depend on accurate, timely information for their financial decisions. You'll collaborate with cross-functional teams including data scientists, product managers, and risk engineers to deliver data solutions that enable real-time fraud detection, personalized recommendations, credit scoring, and regulatory compliance reporting.

Key technical challenges include maintaining data consistency across distributed systems with demanding performance requirements, implementing comprehensive data quality frameworks with real-time validation, optimizing query performance on large datasets, and ensuring complete data lineage and governance across multiple business domains. At Paytm, reliable data streams are fundamental to our operations and our commitment to protecting customers' financial security and maintaining India's digital payments infrastructure.

Key Responsibilities

Data Stream Architecture & Development

• Design and implement reliable, scalable data streams handling high-volume transaction data with strong data integrity controls
• Build real-time processing systems using modern data engineering frameworks (Java/Python stack) with excellent performance characteristics
• Develop robust data ingestion systems from multiple sources with built-in redundancy and monitoring capabilities
• Implement comprehensive data quality frameworks, ensuring the 4 C's: Completeness, Consistency, Conformity, and Correctness - ensuring data reliability that supports sound business decisions
• Design automated data validation, profiling, and quality monitoring systems with proactive alerting capabilities

Infrastructure & Platform Management

• Manage and optimize distributed data processing platforms with high availability requirements to ensure consistent service delivery
• Design data lake and data warehouse architectures with appropriate partitioning and indexing strategies for optimal query performance
• Implement CI/CD processes for data engineering workflows with comprehensive testing and reliable deployment procedures
• Ensure high availability and disaster recovery for critical data systems to maintain business continuity

Performance & Optimization

• Monitor and optimize streaming performance with focus on latency reduction and operational efficiency
• Implement efficient data storage strategies including compression, partitioning, and lifecycle management with cost considerations
• Troubleshoot and resolve complex data streaming issues in production environments with effective response protocols
• Conduct proactive capacity planning and performance tuning to support business growth and data volume increases

Collaboration & Leadership

• Work closely with data scientists, analysts, and product teams to understand important data requirements and service level expectations
• Mentor junior data engineers with emphasis on data quality best practices and customer-focused approach
• Participate in architectural reviews and help establish data engineering standards that prioritize reliability and accuracy
• Document technical designs, processes, and operational procedures with focus on maintainability and knowledge sharing

Required Qualifications

Experience & Education

• Bachelor's or Master's degree in Computer Science, Engineering, or related technical field
• 7+ years (Senior) of hands-on data engineering experience
• Proven experience with large-scale data processing systems (preferably in fintech/payments domain)
• Experience building and maintaining production data streams processing TB/PB scale data with strong performance and reliability standards

Technical Skills & Requirements

• Programming Languages: Expert-level proficiency in both Python and Java
• Big Data Technologies: Apache Spark (PySpark, Spark SQL, Spark with Java), Apache Kafka, Apache Airflow
• Cloud Platforms: AWS (EMR, Glue, Redshift, S3, Lambda) or equivalent Azure/GCP services
• Databases: Strong SQL skills, experience with both relational (PostgreSQL, MySQL) and NoSQL databases (MongoDB, Cassandra, Redis)
• Data Quality Management: Deep understanding of the 4 C's framework - Completeness, Consistency, Conformity, and Correctness
• Data Governance: Experience with data lineage tracking, metadata management, and data cataloging
• Data Formats & Protocols: Parquet, Avro, JSON, REST APIs, GraphQL
• Containerization & DevOps: Docker, Kubernetes, Git, GitLab/GitHub with CI/CD pipeline experience
• Monitoring & Observability: Experience with Prometheus, Grafana, or similar monitoring tools
• Data Modeling: Dimensional modeling, data vault, or similar methodologies
• Infrastructure as Code: Terraform, CloudFormation (preferred)
• Java-specific: Spring Boot, Maven/Gradle, JUnit for building robust data services

Preferred Qualifications

Domain Expertise

• Previous experience in fintech, payments, or banking industry with solid understanding of regulatory compliance and financial data requirements
• Understanding of financial data standards, PCI DSS compliance, and data privacy regulations where compliance is essential for business operations
• Experience with real-time fraud detection or risk management systems where data accuracy is crucial for customer protection

Advanced Technical Skills (Preferred)

• Experience building automated data quality frameworks covering all 4 C's dimensions
• Knowledge of machine learning stream orchestration (MLflow, Kubeflow)
• Familiarity with data mesh or federated data architecture patterns
• Experience with change data capture (CDC) tools and techniques

Leadership & Soft Skills

• Strong problem-solving abilities with experience debugging complex distributed systems in production environments
• Excellent communication skills with ability to explain technical concepts to diverse stakeholders while highlighting business value
• Experience mentoring team members and leading technical initiatives with focus on building a quality-oriented culture
• Proven track record of delivering projects successfully in dynamic, fast-paced financial technology environments

What We Offer

• Opportunity to work with cutting-edge technology at scale
• Competitive salary and equity compensation
• Comprehensive health and wellness benefits
• Professional development opportunities and conference attendance
• Flexible working arrangements
• Chance to impact millions of users across India's digital payments ecosystem

Application Process

Interested candidates should submit:

• Updated resume highlighting relevant data engineering experience with emphasis on real-time systems and data quality
• Portfolio or GitHub profile showcasing data engineering projects, particularly those involving high-throughput streaming systems
• Cover letter explaining interest in fintech/payments domain and understanding of data criticality in financial services
• References from previous technical managers or senior colleagues who can attest to your data quality standards