GCP Data Engineer

Brak informacji o wynagrodzeniu

Nieokreślone0Full-time

#463360·Dodano 25 dni temu·0

Źródło: Xebia

Aplikuj teraz

Tech Stack / Keywords

GCP Dataproc Apache Spark ETL/ELT BigQueryGoogle Cloud StorageSQL Python Apache AirflowPub/Sub

Podsumowanie oferty

You will design, develop, and optimize data pipelines for data platforms on Google Cloud Platform. The work includes ETL/ELT pipelines, Apache Spark workloads, and event-driven orchestration using Pub/Sub and Apache Airflow.

Wymagania

Strong commercial experience building and operating data platforms on Google Cloud Platform.
Hands-on experience with Dataproc and Apache Spark.
Experience designing and developing ETL/ELT pipelines.
Experience with BigQuery and Google Cloud Storage.
Strong SQL and Python skills.
Experience optimizing data-processing performance and cloud costs.
Experience designing ingestion frameworks and reusable data-processing components.
Understanding of data-platform architecture patterns, including Bronze/Silver/Gold layers.
Experience troubleshooting and supporting production data pipelines.
Strong hands-on experience with Apache Airflow.
Experience designing, implementing, and maintaining complex DAGs.

Obowiązki

Design, develop, and optimize ETL/ELT pipelines on Google Cloud Platform.
Optimize Spark workloads running on Dataproc and Dataproc Serverless.
Implement workload-routing strategies and cost-optimization mechanisms.
Design and maintain data-ingestion processes for Bronze, Silver, and Gold layers.
Organize and manage raw-data structures in Google Cloud Storage.
Design and implement event-driven orchestration using Pub/Sub and Apache Airflow.
Develop, maintain, and optimize Airflow DAGs and orchestration frameworks.
Implement data reprocessing and backfill capabilities.
Consolidate related processing workflows into reusable orchestration patterns.
Extend and improve metadata-driven generators and platform tooling.
Support artifact generation, regeneration, and distribution processes.
Improve observability, reliability, and operational support for data pipelines.
Collaborate with platform, analytics, and DevOps teams on architecture and implementation decisions.

Inne informacje

Work from the European Union region and a work permit are required.
Recruitment process: CV review, HR call, interview, client interview, and decision.

Xebia sp. z o.o.

124 aktywne oferty

Zobacz wszystkie oferty

Aplikuj teraz