GCP Data Engineer

Job Description

JOB DESCRIPTION

8+ years of Data Engineering experience

5+ years of Cloud experience (GCP preferred) with solutions designed and implemented at production scale

Strong understanding of key GCP services, especially those related to data processing [Batch/Real Time] Big Query, Cloud Scheduler, Airflow, Postgres, Data Flow, Pub/Sub, Cloud Logging and Cloud Monitoring

Experience with infrastructure as code Terraform/ GitHub

Experience in design, development and implementation of data pipelines using Data Warehousing applications

Experience in integrating various data sources like Oracle, Teradata, DB2, Big Query & Flat files

Hands on experience in performance tuning and debugging ETL jobs

Involving in review meetings and coordinating with the team in job designing and fine-tuning the job performance.

RESPONSIBILITIES

Responsible for gathering business requirements and design ETL systems to meet application needs

Work closely with stakeholders to gather requirements and provide technical guidance on the migration process

Collaborate with the data engineering teams to understand the existing JCL/ETL jobs and their dependencies.

Design and implement migration strategies for moving ETL to GCP native services.

Develop, test, and deploy data pipelines and workflows, ensuring scalability, reliability, and performance

Troubleshoot and resolve any production issues or challenges that arise during the migration

Deliver product features as part of roadmap and business needs

Lead in Effort estimation for the new developments and enhancements.

Co-create test plan and lead execution and code deployment. Lead performance tuning to optimize the job execution.

Responsible for validating, run, scheduling and monitoring jobs using data stage director

Responsible for debugging Ticketing issues in the production phase

QUALIFICATIONS

BE/BTech in CS/IT or equivalent degree in relevant domain

Proven experience in data engineering [IBM Datastage or equivalent Cloud ETL preferred]

Strong knowledge of ETL processes, data integration and data management

Proficiency in programming languages such as SQL, Python or Java

Familiarity with cloud computing platforms and data storage solutions

Excellent problem solving skills and ability to work independently as well as team player

Professional Certification in GCP (e.g., Professional Data Engineer)

Experience in Mainframe to GCP migration considered plus

Agile Methodology Practitioner