Databricks Engineer

6 Months ago • 5 Years + • Software Development & Engineering

Job Summary

Job Description

ShyftLabs is looking for a Databricks Engineer to design, develop, and optimize big data solutions using the Databricks Unified Analytics Platform. The role involves strong expertise in Apache Spark, SQL, Python, and cloud platforms such as AWS, Azure, or GCP. The ideal candidate will collaborate with cross-functional teams to drive data-driven insights and ensure scalable, high-performance data architectures. Responsibilities include designing and implementing big data pipelines, developing ETL workflows, using Apache Spark, and implementing data governance. This position also requires optimizing data lakehouse architectures, collaborating with data scientists, monitoring Databricks clusters, automating workflows, and ensuring data integrity.
Must have:
  • 5+ years of hands-on experience with Databricks and Apache Spark.
  • Proficiency in SQL, Python, or Scala for data processing and analysis.
  • Experience with cloud platforms (AWS, Azure, or GCP) for data engineering.
  • Strong knowledge of ETL frameworks, data lakes, and Delta Lake architecture.
  • Experience with CI/CD tools and DevOps best practices.
  • Familiarity with data security, compliance, and governance best practices.
  • Bachelor's or Master's degree in Computer Science or related field.
Good to have:
  • Databricks certifications (e.g., Databricks Certified Data Engineer).
  • Hands-on experience with MLflow, Feature Store, or Databricks SQL.
  • Exposure to Kubernetes, Docker, and Terraform.
  • Experience with streaming data architectures (Kafka, Kinesis, etc.).
  • Strong understanding of business intelligence and reporting tools.
  • Prior experience working with retail, e-commerce, or ad-tech data platforms.
Perks:
  • Competitive salary
  • Strong insurance package
  • Extensive learning and development resources.

Job Details

Position Overview:

ShyftLabs is seeking a skilled Databricks Engineer to support in designing, developing, and optimizing big data solutions using the Databricks Unified Analytics Platform. This role requires strong expertise in Apache Spark, SQL, Python, and cloud platforms (AWS/Azure/GCP). The ideal candidate will collaborate with cross-functional teams to drive data-driven insights and ensure scalable, high-performance data architectures.


ShyftLabs is a growing data product company that was founded in early 2020 and works primarily with Fortune 500 companies. We deliver digital solutions built to help accelerate the growth of businesses in various industries, by focusing on creating value through innovation.


Job Responsiblities
  • Design, implement, and optimize big data pipelines in Databricks.
  • Develop scalable ETL workflows to process large datasets.
  • Leverage Apache Spark for distributed data processing and real-time analytics.
  • Implement data governance, security policies, and compliance standards.
  • Optimize data lakehouse architectures for performance and cost-efficiency.
  • Collaborate with data scientists, analysts, and engineers to enable advanced AI/ML workflows.
  • Monitor and troubleshoot Databricks clusters, jobs, and performance bottlenecks.
  • Automate workflows using CI/CD pipelines and infrastructure-as-code practices.
  • Ensure data integrity, quality, and reliability in all pipelines.


Basic Qualifications
  • Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
  • 5+ years of hands-on experience with Databricks and Apache Spark.
  • Proficiency in SQL, Python, or Scala for data processing and analysis.
  • Experience with cloud platforms (AWS, Azure, or GCP) for data engineering.
  • Strong knowledge of ETL frameworks, data lakes, and Delta Lake architecture.
  • Experience with CI/CD tools and DevOps best practices.
  • Familiarity with data security, compliance, and governance best practices.
  • Strong problem-solving and analytical skills with an ability to work in a fast-paced environment.


Preferred Qualifications
  • Databricks certifications (e.g., Databricks Certified Data Engineer, Spark Developer).
  • Hands-on experience with MLflow, Feature Store, or Databricks SQL.
  • Exposure to Kubernetes, Docker, and Terraform.
  • Experience with streaming data architectures (Kafka, Kinesis, etc.).
  • Strong understanding of business intelligence and reporting tools (Power BI, Tableau, Looker).
  • Prior experience working with retail, e-commerce, or ad-tech data platforms.


We are proud to offer a competitive salary alongside a strong insurance package. We pride ourselves on the growth of our employees, offering extensive learning and development resources.

Similar Jobs

Scale AI - Senior Product Designer

Scale AI

San Francisco, California, United States (On-Site)
2 Months ago
Wind River - Senior Industry Marketing Manager – Aerospace, Defense, Government

Wind River

Washington, District Of Columbia, United States (On-Site)
1 Month ago
Crunchyroll - Theatrical Film Marketing Manager - EMEA

Crunchyroll

London, England, United Kingdom (Hybrid)
5 Months ago
Riot Games - Technical Operations Program Manager - Esports Event Technology

Riot Games

Los Angeles, California, United States (On-Site)
2 Months ago
Roof Stacks - Scrum Master

Roof Stacks

Istanbul, İstanbul, Türkiye (Hybrid)
3 Months ago
The Walt Disney Company - Senior Software Engineer

The Walt Disney Company

Burbank, California, United States (On-Site)
3 Months ago
eBay - Sr. Software Engineer

eBay

Austin, Texas, United States (On-Site)
2 Months ago
Airbyte - Software Engineer, Deployments

Airbyte

San Francisco, California, United States (On-Site)
3 Months ago
Apple - VLSI CAD Engineer

Apple

Austin, Texas, United States (On-Site)
1 Month ago
NVIDIA - Senior Mixed Signal Circuit Design Engineer

NVIDIA

Hsinchu, Hsinchu City, Taiwan (On-Site)
6 Months ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Patreon - Senior Data Scientist

Patreon

San Francisco, California, United States (Hybrid)
4 Months ago
Blinkhealth - Principal Product Manager

Blinkhealth

New York, United States (On-Site)
2 Months ago
ElevenLabs - Sales Enablement

ElevenLabs

San Francisco, California, United States (Remote)
4 Months ago
Paytm - Growth Management- AGM/ GM- EDC

Paytm

Noida, Uttar Pradesh, India (On-Site)
2 Months ago
Qualcomm - Senior Automotive Platform/BSP/Safety Application Engineer

Qualcomm

Ingolstadt, Bavaria, Germany (On-Site)
2 Months ago
onwards Search - Marketing Analyst

onwards Search

Plantation, Florida, United States (Hybrid)
2 Months ago
ShyftLabs - Technical Project Manager

ShyftLabs

Toronto, Ontario, Canada (Hybrid)
2 Months ago
NCR Voyix - Inventory Revenue Accounting Analyst -II

NCR Voyix

India (On-Site)
1 Month ago
NVIDIA - Principal Thermal Mechanical Photonic Designer

NVIDIA

Santa Clara, California, United States (On-Site)
6 Months ago
Accurate - Cloud Engineering Architect

Accurate

United States (Remote)
1 Month ago

Get notifed when new similar jobs are uploaded

Jobs in Toronto, Ontario, Canada

GoDaddy - Senior Paralegal

GoDaddy

British Columbia, Canada (Remote)
2 Months ago
NVIDIA - Senior Photonic Layout Design Engineer

NVIDIA

Canada (On-Site)
5 Months ago
Rippling - Senior Growth Manager

Rippling

Toronto, Ontario, Canada (On-Site)
3 Months ago
Qualcomm - GPU Design Verification Engineer

Qualcomm

Markham, Ontario, Canada (On-Site)
2 Months ago
Ubisoft - Senior Marketing Analyst

Ubisoft

Montreal, Quebec, Canada (On-Site)
3 Months ago
Nagarro - Senior Staff Engineer, PHP Magento

Nagarro

Canada (Remote)
9 Months ago
Ion - Technical Support Analyst, Toronto - 4363

Ion

Toronto, Ontario, Canada (On-Site)
9 Months ago
Amber - Localization Quality Assurance (Korean)

Amber

Quebec, Canada (Hybrid)
4 Months ago
Ubisoft - Lead Technical Rainbow Six

Ubisoft

Montreal, Quebec, Canada (On-Site)
4 Months ago
Ubisoft - Accountant

Ubisoft

Montreal, Quebec, Canada (On-Site)
3 Months ago

Get notifed when new similar jobs are uploaded

Software Development & Engineering Jobs

Escape Velocity Entertainment - Release Engineer

Escape Velocity Entertainment

(Remote)
4 Months ago
rivos - SOC Electrical Analysis Engineer - Full Time

rivos

Bengaluru, Karnataka, India (Hybrid)
9 Months ago
Philips - Field Service Engineer, Medical Imaging

Philips

Denver, Colorado, United States (On-Site)
1 Month ago
PwC - Associate-SAP FICO-Kolkata-RDC

PwC

Kolkata, West Bengal, India (On-Site)
10 Months ago
Crunchyroll - Principal Software Engineer, Video Players

Crunchyroll

Dallas, Texas, United States (On-Site)
6 Months ago
Apple - Circuits Implementation Engineer

Apple

Austin, Texas, United States (On-Site)
1 Month ago
gesture talk studios - Senior Software Engineer

gesture talk studios

Canada (On-Site)
1 Month ago
Hawkeye Innovations - 3rd Line Support Engineer

Hawkeye Innovations

London, England, United Kingdom (Hybrid)
4 Months ago
PwC - Associate - SAP- GDC

PwC

Kolkata, West Bengal, India (On-Site)
10 Months ago
Apple - Wireless/Telephony Software Engineer

Apple

San Diego, California, United States (On-Site)
2 Months ago

Get notifed when new similar jobs are uploaded

About The Company

Here at ShyftLabs, we build data products to help enterprises deliver real impact through tailored data analytics, science, and AI solutions. From consulting to operations, we guide our customers through their data journey and ensure they are data and AI-empowered.

Noida, Uttar Pradesh, India (On-Site)

Noida, Uttar Pradesh, India (Hybrid)

Noida, Uttar Pradesh, India (Hybrid)

Noida, Uttar Pradesh, India (Hybrid)

Knoxville, Tennessee, United States (Hybrid)

Noida, Uttar Pradesh, India (Hybrid)

Noida, Uttar Pradesh, India (Hybrid)

Noida, Uttar Pradesh, India (On-Site)

Noida, Uttar Pradesh, India (Hybrid)

Noida, Uttar Pradesh, India (Hybrid)

View All Jobs

Get notified when new jobs are added by ShyftLabs

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug