Senior Data Engineer

JOB OVERVIEW

Plan A Technologies is looking for a highly skilled Senior Data Engineer with extensive experience in Databricks, including Unity Catalog for data governance. The ideal candidate will possess a strong ability to analyze and optimize cluster configurations using cluster logs and demonstrate a proactive approach to independently drive tasks to completion.

Please note: you must have at least 6+ years of experience in data engineering, with a strong focus on Databricks and big data processing to be considered for this role.

JOB RESPONSIBILITY

• Design, develop, and maintain scalable data pipelines and workflows within the Databricks environment, ensuring optimal performance and cost-efficiency.

• Unity Catalog Implementation: Deploy and manage Unity Catalog to provide centralized data governance, access management, and auditing across Databricks environments.

• Analyze cluster logs to fine-tune configurations, optimize resource allocation, and enhance overall system performance.

• Take ownership of well-defined tasks, driving them to completion with minimal supervision, and proactively addressing potential challenges.

• Work closely with cross-functional teams, including data scientists, analysts, and other engineers, to understand data requirements and deliver robust solutions.

• Maintain comprehensive documentation of data workflows, configurations, and best practices to support knowledge sharing and onboarding.

EXPERIENCE

• BS in Computer Science, Software Engineering or equivalent practical experience

• 6+ years of experience in data engineering, with a strong focus on Databricks and big data processing.

• Proficiency in Databricks, including Unity Catalog for data governance.

• Strong experience with cluster configuration analysis and optimization using cluster logs.

• Expertise in programming languages such as Python or Scala.

• Solid understanding of data modeling, ETL processes, and data warehousing concepts.

• Experience with cloud platforms such as AWS or Azure.

• Familiarity with CI/CD pipelines and DevOps practices.

• Knowledge of data security and compliance standards.

• Nice to have: Certifications related to Databricks or cloud platforms

• Demonstrated ability to work independently and take initiative.

• Excellent verbal and written English communication skills.

• Excellent problem-solving and analytical skills.

• Strong communication and interpersonal abilities.

• Initiative and drive to do great things.

ABOUT THE COMPANY/BENEFITS

Plan A Technologies is an American software development and technology advisory firm that brings top-tier engineering talent to clients around the world. Our software engineers tackle custom product development projects, staff augmentation, major integrations and upgrades, and much more. The team is far more hands-on than the giant outsourcing shops, but still big enough to handle major enterprise clients.

Read more about us here: www.PlanAtechnologies.com .

Location: Work From Home 100% of the time, or come in to one of our global offices. Up to you.

Great colleagues and an upbeat work environment: You'll join an excellent team of supportive engineers and project managers who work hard but don't ever compete with each other.

Benefits: You’ll get a generous vacation schedule, Brand New Laptop, and other goodies.

If this sounds like you, we'd love to hear from you!