Staff Software Engineer, Data Ingestion
Job Summary
Job Description
The Staff Software Engineer, Data Ingestion is a crucial individual contributor responsible for designing data collection strategies and developing/maintaining robust, scalable data pipelines. This role supports the data ecosystem by delivering analytical software solutions for timely, accurate, and complete data access to drive insights, products, and operational efficiency. Key responsibilities include designing and developing high-performance, fault-tolerant ingestion pipelines using Python, integrating with diverse data sources like APIs and streaming platforms, implementing data transformations, and monitoring/troubleshooting pipelines. Collaboration with database engineers for data model optimization and evaluation of new technologies for ingestion improvement are also key aspects of this role.
Must have:
- 6+ years in software development
- Extensive Python expertise
- Data collection & integration experience
- Understanding of distributed systems
- Cloud platform experience (AWS/GCP)
- Database fundamentals (SQL)
- Monitoring & alerting experience
- Git proficiency
Good to have:
- Containerization (Docker/Kubernetes)
- Streaming technologies (Kafka, Flink, Spark Streaming)
- OLAP database experience (Hadoop)
13 skills required
Job Details
The Staff Software Engineer, Data Ingestion will be a critical individual contributor responsible for designing collection strategies, developing, and maintaining robust and scalable data pipelines. This role is at the heart of our data ecosystem, deliver new analytical software solution to access timely, accurate, and complete data for insights, products, and operational efficiency.
Key Responsibility
- Design, develop, and maintain high-performance, fault-tolerant data ingestion pipelines using Python.
- Integrate with diverse data sources (databases, APIs, streaming platforms, cloud storage, etc.).
- Implement data transformation and cleansing logic during ingestion to ensure data quality.
- Monitor and troubleshoot data ingestion pipelines, identifying and resolving issues promptly.
- Collaborate with database engineers to optimize data models for fast consumption.
- Evaluate and propose new technologies or frameworks to improve ingestion efficiency and reliability.
- Develop and implement self-healing mechanisms for data pipelines to ensure continuity.
- Define and uphold SLAs and SLOs for data freshness, completeness, and availability.
- Participate in on-call rotation as needed for critical data pipeline issues.
Required Skills
- 6+ years experience in software development industry from computer science background
- Extensive Python Expertise: Extensive experience in developing robust, production-grade applications with Python.
- Data Collection & Integration: Proven experience collecting data from various sources (REST APIs, OAuth, GraphQL, Kafka, S3, SFTP, etc.).
- Distributed Systems & Scalability: Strong understanding of distributed systems concepts, designing for scale, performance optimization, and fault tolerance.
- Cloud Platforms: Experience with major cloud providers (AWS or GCP) and their data-related services (e.g., S3, EC2, Lambda, SQS, Kafka, Cloud Storage, GKE).
- Database Fundamentals: Solid understanding of relational databases (SQL, schema design, indexing, query optimization). OLAP database experience is a plus (Hadoop)
- Monitoring & Alerting: Experience with monitoring tools (e.g., Prometheus, Grafana) and setting up effective alerts.
- Version Control: Proficiency with Git.
- Containerization (Plus): Experience with Docker and Kubernetes.
- Streaming Technologies (Plus): Experience with real-time data processing using Kafka, Flink, Spark Streaming
Similar Jobs
Brave Group (Language barrier)
Kawasaki, Kanagawa, Japan (Hybrid)
• 2 Months ago
Barracuda
Bengaluru, Karnataka, India (On-Site)
• 6 Months ago
Nagarro
Allentown, Pennsylvania, United States (Remote)
• 9 Months ago
Get notifed when new similar jobs are uploaded
Similar Skill Jobs
Shield AI
San Diego, California, United States (On-Site)
• 1 Week ago
Get notifed when new similar jobs are uploaded
Jobs in Hyderabad, Telangana, India
CME Group
Bengaluru, Karnataka, India (On-Site)
• 2 Weeks ago
Get notifed when new similar jobs are uploaded
Data Analysis Jobs
bytedance
San Jose, California, United States (On-Site)
• 4 Months ago
Valeo
Tangier, Tangier-Tétouan-Al Hoceima, Morocco (On-Site)
• 1 Week ago
Discord
San Francisco, California, United States (On-Site)
• 4 Weeks ago
Get notifed when new similar jobs are uploaded
