Job Description
As a Lead Big Data Engineer, you will combine hands-on engineering with technical leadership, designing, developing, and optimizing Spark-based big data pipelines in Palantir Foundry for high performance and scalability. You will also mentor and manage a team, driving best practices, ensuring delivery excellence, and collaborating with stakeholders to meet business needs. Prior Palantir Foundry experience is a plus, but not mandatory.
Good To Have:
- Experience with Palantir Foundry.
- Familiarity with CI/CD.
- Familiarity with infrastructure as code (Terraform, CloudFormation).
Must Have:
- Lead the design, development, and optimization of large-scale, Spark-based (PySpark) data processing pipelines.
- Build and maintain big data solutions using Palantir Foundry.
- Ensure Spark workloads are tuned for performance and cost efficiency.
- Oversee and participate in code reviews, architecture discussions, and best practice implementation.
- Maintain high standards for data quality, security, and governance.
- Manage and mentor a team of Big Data Engineers, providing technical direction.
- Drive continuous improvement in processes, tools, and development practices.
- Foster collaboration across engineering, data science, and product teams to align on priorities and solutions.
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
- 6+ years in Big Data Engineering, with at least 1-2 years in a lead (tech/team lead) role.
- Deep hands-on expertise in Apache Spark (PySpark) for large-scale data processing.
- Proficiency in Python and distributed computing principles.
- Experience designing, implementing, and optimizing high-volume, low-latency data pipelines.
- Strong leadership, communication, and stakeholder management skills.
Perks:
- Flexible working format - remote, office-based or flexible.
- A competitive salary and good compensation package.
- Personalized career growth.
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more).
- Active tech communities with regular knowledge sharing.
- Education reimbursement.
- Memorable anniversary presents.
- Corporate events and team buildings.
- Other location-specific benefits.
Role Overview:
As a Lead Big Data Engineer, you will combine hands-on engineering with technical leadership. You’ll be responsible for designing, developing, and optimizing Spark-based big data pipelines in Palantir Foundry, ensuring high performance, scalability, and reliability. You will also mentor and manage a team of engineers, driving best practices in big data engineering, ensuring delivery excellence, and collaborating with stakeholders to meet business needs. While our project uses Palantir Foundry, prior experience with it is a plus, but not mandatory.
Key Responsibilities:
- Lead the design, development, and optimization of large-scale, Spark-based (PySpark) data processing pipelines.
- Build and maintain big data solutions using Palantir Foundry
- Ensure Spark workloads are tuned for performance and cost efficiency.
- Oversee and participate in code reviews, architecture discussions, and best practice implementation.
- Maintain high standards for data quality, security, and governance.
- Manage and mentor a team of Big Data Engineers, providing technical direction
- Drive continuous improvement in processes, tools, and development practices.
- Foster collaboration across engineering, data science, and product teams to align on priorities and solutions.
Requirements:
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
- 6+ years in Big Data Engineering, with at least 1-2 years in a lead (tech/team lead) role.
- Deep hands-on expertise in Apache Spark (PySpark) for large-scale data processing.
- Proficiency in Python and distributed computing principles.
- Experience designing, implementing, and optimizing high-volume, low-latency data pipelines.
- Strong leadership, communication, and stakeholder management skills.
- Experience with Palantir Foundry is a plus, but not required.
- Familiarity with CI/CD and infrastructure as code (Terraform, CloudFormation) is desirable.
We offer*:
- Flexible working format - remote, office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
*not applicable for freelancers
Set alerts for more jobs like Senior/Lead Big Data Engineer + AI
Set alerts for new jobs by N-ix
Set alerts for new Data Analysis jobs in Ukraine
Set alerts for new jobs in Ukraine
Set alerts for Data Analysis (Remote) jobs
