Lead Data Engineer - R01553054

1 Minute ago • 5 Years + • Data Analysis

Job Summary

Job Description

This role is for a Senior Data Engineer with deep expertise in Google Cloud Platform (GCP) and BigQuery, focusing on leading cloud modernization initiatives. Key responsibilities include migrating large-scale datasets to Google BigQuery, designing data migration strategies, integrating data from various sources, building real-time streaming pipelines, and modernizing ETL processes. The role also involves developing scalable workflows using Apache Airflow, Python, Spark, and GCP-native tools, as well as optimizing SQL queries and transformation scripts for high-performance data processing.
Must have:
  • Analyze legacy on-premises and hybrid cloud data warehouse environments.
  • Lead the migration of large-scale datasets to Google BigQuery.
  • Design and implement data migration strategies ensuring data quality, integrity, and performance.
  • Integrate data from various structured and unstructured sources.
  • Build real-time streaming pipelines for large-scale ingestion and processing of IoT and telemetry data.
  • Modernize and refactor legacy SSIS packages into cloud-native ETL pipelines.
  • Develop scalable, reliable workflows using Apache Airflow, Python, Spark, and GCP-native tools.
  • Ensure high-performance data transformation and loading into BigQuery for analytical use cases.
  • Write and optimize complex SQL queries, stored procedures, and scheduled jobs within BigQuery.
  • Develop modular, reusable transformation scripts using Python, Java, Spark, and SQL.
  • Continuously monitor and optimize query performance and cost efficiency in the cloud data environment.
Good to have:
  • GCP Professional Data Engineer certification
  • dbt
  • Kafka
  • Terraform
  • ML pipelines
  • analytics engineering
  • DataOps/DevOps methodologies
Perks:
  • Work with cutting-edge technologies in a fast-paced, collaborative environment.
  • Lead cloud transformation initiatives at scale.

Job Details

Primary Skills

  • IICS, Alation, Data Modelling Fundamentals, Data Warehousing, ETL Fundamentals, Modern Data Platform Fundamentals, PLSQL, T-SQL, Stored Procedures, Python, SQL, SQL (Basic + Advanced), Talend

Job requirements

  • About the Role
  • We are seeking a Senior Data Engineer with deep expertise in Google Cloud Platform (GCP) and BigQuery to lead cloud modernization initiatives, develop scalable data pipelines, and enable real-time data processing for enterprise-level systems. This is a high-impact role focused on driving the transformation of legacy infrastructure into a robust, cloud-native data ecosystem.

*

  • Key Responsibilities
  • 1. Data Migration & Cloud Modernization
  • Analyze legacy on-premises and hybrid cloud data warehouse environments (e.g., SQL Server).
  • Lead the migration of large-scale datasets to Google BigQuery.
  • Design and implement data migration strategies ensuring data quality, integrity, and performance.
  • 2. Data Integration & Streaming
  • Integrate data from various structured and unstructured sources, including APIs, relational databases, and IoT devices.
  • Build real-time streaming pipelines for large-scale ingestion and processing of IoT and telemetry data.
  • 3. ETL / Data Pipeline Development
  • Modernize and refactor legacy SSIS packages into cloud-native ETL pipelines.
  • Develop scalable, reliable workflows using Apache Airflow, Python, Spark, and GCP-native tools.
  • Ensure high-performance data transformation and loading into BigQuery for analytical use cases.
  • 4. Programming & Query Optimization
  • Write and optimize complex SQL queries, stored procedures, and scheduled jobs within BigQuery.
  • Develop modular, reusable transformation scripts using Python, Java, Spark, and SQL.
  • Continuously monitor and optimize query performance and cost efficiency in the cloud data environment.

*

  • Required Skills & Experience
  • 5+ years in Data Engineering with a strong focus on cloud and big data technologies.
  • Minimum 2+ years of hands-on experience with GCP, specifically BigQuery.
  • Proven experience migrating on-premise data systems to the cloud.
  • Strong development experience with Apache Airflow, Python, and Apache Spark.
  • Expertise in streaming data ingestion, particularly in IoT or sensor data environments.
  • Strong SQL development skills; experience with BigQuery performance tuning.
  • Solid understanding of cloud architecture, data modeling, and data warehouse design.
  • Familiarity with Git and CI/CD practices for managing data pipelines.

*

  • Preferred Qualifications
  • GCP Professional Data Engineer certification.
  • Experience with modern data stack tools like dbt, Kafka, or Terraform.
  • Exposure to ML pipelines, analytics engineering, or DataOps/DevOps methodologies.
  • Why Join Us?
  • Work with cutting-edge technologies in a fast-paced, collaborative environment.
  • Lead cloud transformation initiatives at scale.

Similar Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

Similar Skill Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

Jobs in Bengaluru, Karnataka, India

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

Data Analysis Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

About The Company

Phoenix, Arizona, United States (Remote)

Bengaluru, Karnataka, India (Hybrid)

Jersey City, New Jersey, United States (On-Site)

London, England, United Kingdom (Hybrid)

Chicago, Illinois, United States (Hybrid)

Austin, Texas, United States (Remote)

Irvine, California, United States (On-Site)

Bengaluru, Karnataka, India (Hybrid)

Bengaluru, Karnataka, India (Hybrid)

Bengaluru, Karnataka, India (Hybrid)

View All Jobs

Get notified when new jobs are added by Brillio

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug