GCP Data Engineer
Job Summary
Job Description
The GCP Data Engineer is responsible for building and developing large-scale cloud data processing systems on the Google Cloud Platform (GCP). This role requires significant expertise in data warehousing and proven coding skills in Python, Java, SQL, and Spark. The engineer will implement enterprise cloud data architecture designs, collaborating with scrum teams and business partners to identify, evaluate, design, and implement solutions for structured and unstructured data. The primary focus is on iteratively developing scalable, high-performance solutions that deliver measurable business value on the cloud platform.
Must have:
- Minimum 10+ years of work experience
- 5+ years in engineering role with Python, Java, Spark, SQL
- 5+ years as Data Engineer in GCP
- Proficiency with Google's IAM API
- Proficiency with Airflow
Good to have:
- GCP Data Engineer certification
- Coding experience with Python, Java, Spark, SQL
- Linux/Unix background
- Big data technologies (HDFS, Spark, Impala, Hive)
- GCP platform tools (Pub/sub, Cloud Storage, Big Table, Big Query, Data Flow, Data Proc, Composer)
- Hadoop and cloud platforms knowledge
- Web services and APIs (RESTful, SOAP)
- Real-time streaming and batch processing (Spark, Kafka, Flume, Pub/sub, Airflow)
- Experience with Avro, Parquet, JSON
- Shell scripting and bash
- Version control (GitHub)
- Unit testing code
- Development ecosystem (Jenkins, Artifactory, CI/CD, Terraform)
- API Orchestration and Choreography
- Apache packages and Hybrid cloud architectures
- Pipeline creation and automation for Data Acquisition
- Metadata extraction pipeline design
- Quality control metrics for data acquisition
- Experience with Jira and Confluence
- Managing and scheduling batch jobs
- SDLC experience (Analysis, Design, Coding, Testing)
- Scala coding skills
- Ability to document designs and concepts
23 skills required
Job Details
Description
GCP Data Engineer
Responsible for construction and development of "large-scale cloud data processing systems" in the Google Cloud Platform (GCP). The GCP Data Engineer must have considerable expertise in data warehousing and the job requires proven coding expertise with Python, Java, SQL, and Spark languages. Must be able to implement enterprise cloud data architecture designs, and will work closely with the rest of the scrum team and internal business partners to identify, evaluate, design, and implement large scale data solutions, structured and unstructured, public and proprietary data. The GCP Data Engineer will work iteratively on the cloud platform to design, develop and implement scalable, high performance solutions that offer measurable business value to customers.
Qualifications and Education:
- GCP Data Engineer certification preferred
- Bachelor's in computer engineering or equivalent field or equivalent foreign degree required
Required Work Experience:
- Minimum of 10+ years of work experience
- 5+ years of experience in an engineering role using Python, Java, Spark, and SQL.
- 5+ experience working as a Data Engineer in GCP
- Demonstrated proficiency with Google’s Identity and Access Management (IAM) API
- Demonstrated proficiency with Airflow
Desired Work Experience:
- Coding experience with Python, Java, Spark, and SQL
- Strong Linux/Unix background and hands on knowledge.
- Past experience with big data technologies including HDFS, Spark, Impala, Hive
- Experience with gcp platform development tools Pub/sub, cloud storage, big table, big query, data flow, data proc, and composer desired.
- Knowledge in Hadoop and cloud platforms and surrounding ecosystems.
- Experience with web services and APIs as in RESTful and SOAP.
- Strong experience working with real time streaming applications and batch style large scale distributed computing applications using tools like Spark, Kafka, Flume, pubsub, and airflow.
- Ability to work with different file formats like Avro, Parquet, and JSON.
- Experience with Shell scripting and bash.
- Experience with version control platform github
- Experience unit testing code.
- Experience with development ecosystem including Jenkins, Artifactory, CI/CD, and Terraform.
- Works on problems of diverse scope and complexity ranging from moderate to substantial
- Assists senior professionals in determining methods and procedures for new tasks
- Leads basic or moderately complex projects/activities on semi-regular basis
- Must possess excellent written and verbal communication skills
- Ability to understand and analyze complex data sets
- Exercises independent judgment on basic or moderately complex issues regarding job and related tasks
- Makes recommendations to management on new processes, tools and techniques, or development of new products and services
- Makes decisions regarding daily priorities for a work group; provides guidance to and/or assists staffon non-routine or escalated issues
- Decisions have a moderate impact on operations within a department
- Works under minimal supervision, uses independent judgment requiring analysis of variable factors
- Requires little instruction on day-to-day work and general direction on more complex tasks and projects
- Collaborates with senior professionals in the development of methods, techniques and analytical approach
- Ability to advise management on approaches to optimize for data platform success.
- Able to effectively communicate highly technical information to numerous audiences, including management, the user community, and less-experienced staff.
- Consistently communicate on status of project deliverables
- Consistently provide work effort estimates to management to assist in setting priorities
- Deliver timely work in accordance with estimates
- Solve problems as they arise and communicate potential roadblocks to manage expectations
- Adhere strictly to all security policies
- Proficient in multiple programming languages, frameworks, domains, and tools.
- Coding skills in Scala
- Ability to document designs and concepts
- API Orchestration and Choreography for consumer apps
- Well rounded technical expertise in Apache packages and Hybrid cloud architectures
- Pipeline creation and automation for Data Acquisition
- Metadata extraction pipeline design and creation between raw and finally transformed datasets
- Quality control metrics data collection on data acquisition pipelines
- Able to collaborate with scrum team including scrum master, product owner, data analysts, Quality Assurance, business owners, and data architecture to produce the best possible end products
- Experience contributing to and leveraging jira and confluence.
- Managing and scheduling batch jobs.
- Hands on experience in Analysis, Design, Coding and Testing phases of Software Development Life Cycle (SDLC).
Similar Jobs
Looks like we're out of matches
Set up an alert and we'll send you similar jobs the moment they appear!
Similar Skill Jobs
Looks like we're out of matches
Set up an alert and we'll send you similar jobs the moment they appear!
Jobs in United States
Looks like we're out of matches
Set up an alert and we'll send you similar jobs the moment they appear!
Data Analysis Jobs
Looks like we're out of matches
Set up an alert and we'll send you similar jobs the moment they appear!
