Staff Software Engineer, Data Ingestion

3 Weeks ago • 6 Years + • Data Analysis

Job Summary

Job Description

The Staff Software Engineer, Data Ingestion is a crucial individual contributor responsible for designing data collection strategies and developing/maintaining robust, scalable data pipelines. This role supports the data ecosystem by delivering analytical software solutions for timely, accurate, and complete data access to drive insights, products, and operational efficiency. Key responsibilities include designing and developing high-performance, fault-tolerant ingestion pipelines using Python, integrating with diverse data sources like APIs and streaming platforms, implementing data transformations, and monitoring/troubleshooting pipelines. Collaboration with database engineers for data model optimization and evaluation of new technologies for ingestion improvement are also key aspects of this role.
Must have:
  • 6+ years in software development
  • Extensive Python expertise
  • Data collection & integration experience
  • Understanding of distributed systems
  • Cloud platform experience (AWS/GCP)
  • Database fundamentals (SQL)
  • Monitoring & alerting experience
  • Git proficiency
Good to have:
  • Containerization (Docker/Kubernetes)
  • Streaming technologies (Kafka, Flink, Spark Streaming)
  • OLAP database experience (Hadoop)

Job Details

The Staff Software Engineer, Data Ingestion will be a critical individual contributor responsible for designing collection strategies, developing, and maintaining robust and scalable data pipelines. This role is at the heart of our data ecosystem, deliver new analytical software solution to access timely, accurate, and complete data for insights, products, and operational efficiency.

Key Responsibility

  • Design, develop, and maintain high-performance, fault-tolerant data ingestion pipelines using Python.
  • Integrate with diverse data sources (databases, APIs, streaming platforms, cloud storage, etc.).
  • Implement data transformation and cleansing logic during ingestion to ensure data quality.
  • Monitor and troubleshoot data ingestion pipelines, identifying and resolving issues promptly.
  • Collaborate with database engineers to optimize data models for fast consumption.
  • Evaluate and propose new technologies or frameworks to improve ingestion efficiency and reliability.
  • Develop and implement self-healing mechanisms for data pipelines to ensure continuity.
  • Define and uphold SLAs and SLOs for data freshness, completeness, and availability.
  • Participate in on-call rotation as needed for critical data pipeline issues.

Required Skills

  • 6+ years experience in software development industry from computer science background
  • Extensive Python Expertise: Extensive experience in developing robust, production-grade applications with Python.
  • Data Collection & Integration: Proven experience collecting data from various sources (REST APIs, OAuth, GraphQL, Kafka, S3, SFTP, etc.).
  • Distributed Systems & Scalability: Strong understanding of distributed systems concepts, designing for scale, performance optimization, and fault tolerance.
  • Cloud Platforms: Experience with major cloud providers (AWS or GCP) and their data-related services (e.g., S3, EC2, Lambda, SQS, Kafka, Cloud Storage, GKE).
  • Database Fundamentals: Solid understanding of relational databases (SQL, schema design, indexing, query optimization). OLAP database experience is a plus (Hadoop)
  • Monitoring & Alerting: Experience with monitoring tools (e.g., Prometheus, Grafana) and setting up effective alerts.
  • Version Control: Proficiency with Git.
  • Containerization (Plus): Experience with Docker and Kubernetes.
  • Streaming Technologies (Plus): Experience with real-time data processing using Kafka, Flink, Spark Streaming

Similar Jobs

Casumo - Backend Engineer

Casumo

Skopje, Greater Skopje, North Macedonia (Hybrid)
2 Days ago
binance - Smart Contract Security Engineer

binance

Dubai, Dubai, United Arab Emirates (Remote)
1 Year ago
endava - Test Automation Engineer

endava

Sofia, Sofia City Province, Bulgaria (On-Site)
2 Weeks ago
Match Group - Senior Software Engineer, Backend

Match Group

West Hollywood, California, United States (Hybrid)
1 Week ago
Syniverse - Sr Quality Assurance

Syniverse

Bengaluru, Karnataka, India (Hybrid)
1 Month ago
Lionbridge Games - Business Analyst

Lionbridge Games

Mexico City, Mexico (On-Site)
1 Month ago
Next Level Business Services - Data Scientist -  Full Time Only

Next Level Business Services

Redmond, Washington, United States (On-Site)
8 Months ago
Fanatee - Product Business Analyst

Fanatee

São Paulo, State Of São Paulo, Brazil (Hybrid)
4 Months ago
Head Digital Works - Business Intelligence Analyst

Head Digital Works

Hyderabad, Telangana, India (On-Site)
8 Months ago
Scopely - Data Science Manager, Marketing Analytics

Scopely

Barcelona, Catalonia, Spain (Hybrid)
4 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Crowd Strick - Sr. Engineer - Sensor SDET

Crowd Strick

India (Remote)
1 Year ago
ALTEN - Full Stack Java / JavaScript Developer – Backend & Frontend

ALTEN

Rabat, Rabat-Salé-Kénitra, Morocco (Remote)
1 Week ago
Epic Games - Senior SDET

Epic Games

Montreal, Quebec, Canada (On-Site)
3 Months ago
ISS Stoxx - Senior Software Engineer in C#/.NET and AWS

ISS Stoxx

Mumbai, Maharashtra, India (On-Site)
2 Weeks ago
Brain Hunter - Duck Creek Developer

Brain Hunter

Jersey City, New Jersey, United States (On-Site)
7 Years ago
N-ix - Senior Java Engineer

N-ix

Ukraine (On-Site)
5 Days ago
Morning Star - Lead Software Engineer, Credit Technology

Morning Star

Mumbai, Maharashtra, India (Hybrid)
8 Months ago
Ion - Lead Software Engineer, Italy

Ion

Turin, Piedmont, Italy (On-Site)
8 Months ago
Rockstar Games - Associate Principal Analytics Engineer

Rockstar Games

Edinburgh, Scotland, United Kingdom (On-Site)
2 Months ago
Qualcomm - Engineer - FOSS (Free and Open-Source Software)

Qualcomm

Hyderabad, Telangana, India (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in Hyderabad, Telangana, India

techholding - Varicent ICM Engineer

techholding

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Trellix - Agile Coach

Trellix

Bengaluru, Karnataka, India (On-Site)
1 Month ago
Nagarro - Senior Engineer, Mainframe

Nagarro

India (Remote)
8 Months ago
Trellix - Software Development Engineer Fullstack

Trellix

India (Remote)
3 Weeks ago
Hitachi - Data Scientist

Hitachi

Pune, Maharashtra, India (Remote)
8 Months ago
PayPal - Manager, Software Development

PayPal

Chennai, Tamil Nadu, India (Hybrid)
2 Weeks ago
PwC - Senior Associate - Data Engineer - D&AT IFS

PwC

Bengaluru, Karnataka, India (On-Site)
9 Months ago
Capgemini - Playwright Test Automation

Capgemini

Mumbai, Maharashtra, India (On-Site)
3 Months ago
Paytm - Recruiter - Sales (Bulk Hiring)

Paytm

Noida, Uttar Pradesh, India (On-Site)
3 Months ago
Wind River - Director, Engineering

Wind River

Chennai, Tamil Nadu, India (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

Data Analysis Jobs

The Walt Disney Company - Business Systems Analyst

The Walt Disney Company

Hong Kong (On-Site)
3 Months ago
ShyftLabs - Lead Data Engineer

ShyftLabs

Noida, Uttar Pradesh, India (Hybrid)
4 Weeks ago
Match Group - Sr. Data & Analytics Engineer

Match Group

Seoul, South Korea (Hybrid)
1 Month ago
CyberArk - Data Architect

CyberArk

Israel (Hybrid)
1 Month ago
GoMotive - Senior Data Analyst

GoMotive

United States (Remote)
2 Months ago
Applike - Senior Data Scientist

Applike

Hamburg, Hamburg, Germany (Hybrid)
5 Months ago
Brillio - Data Science Lead (ML/AI)

Brillio

Chicago, Illinois, United States (Hybrid)
2 Weeks ago
Tide - Senior Data Engineer (DBT/Snowflake)

Tide

Delhi, India (Hybrid)
1 Month ago
Razer - Senior Data Scientist – AI Integration

Razer

Singapore (On-Site)
2 Weeks ago
Tencent - Security Data Engineer

Tencent

Kuala Lumpur, Federal Territory Of Kuala Lumpur, Malaysia (On-Site)
1 Month ago

Get notifed when new similar jobs are uploaded

About The Company

Hyderabad, Telangana, India (Remote)

Chicago, Illinois, United States (On-Site)

Cleveland, Ohio, United States (On-Site)

Cleveland, Ohio, United States (On-Site)

Cleveland, Ohio, United States (On-Site)

Chicago, Illinois, United States (On-Site)

Cleveland, Ohio, United States (On-Site)

London, England, United Kingdom (On-Site)

Cleveland, Ohio, United States (On-Site)

Hyderabad, Telangana, India (Remote)

View All Jobs

Get notified when new jobs are added by Bright Edge

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug