Capgemini

(Senior) Databricks Engineer

Brak informacji o wynagrodzeniu

SeniorHybryda·WarszawaFull-time

#463349·Dodano 24 dni temu·1

Źródło: Capgemini

Aplikuj teraz

Tech Stack / Keywords

Databricks Spark SQL PythonDelta LakeLakeflowTerraformUnity CatalogMLflow Azure

Podsumowanie oferty

You will build end-to-end lakehouse platforms on Databricks for clients, covering platform setup, ingestion, pipelines, data models and operations. The work includes data products for analytics and machine learning, using Spark, SQL, Delta Lake and Terraform.

Wymagania

Engineer: 3+ years in data engineering, including at least one year hands-on with Databricks in production. Senior Engineer: 5+ years, including at least three years on Databricks, hands-on work in at least three specified areas, and a track record of leading the technical side of a workstream.
Strong Spark (PySpark or Scala) and SQL skills; Python, Delta Lake, Lakeflow pipelines and jobs, and Databricks SQL.
Working knowledge of one cloud platform (Azure preferred; AWS or GCP), including storage, identity and networking as they apply to Databricks.
Git and CI/CD; experience with Databricks Asset Bundles or Terraform (both at senior level).
Databricks Certified Data Engineer Associate certification held or in progress (Professional certification at senior level).
Unity Catalog basics, including catalogs, permissions and lineage.
Clear communication with client data teams and fluent English; at senior level, ability to present a technical design to client architects.

Nice to have:

Interest in data governance.
Experience in financial services, energy, pharma or the public sector.

Obowiązki

Set up and configure Databricks platforms, including workspaces, compute policies, Unity Catalog basics, networking and identity; at senior level, own platform design and its cost model.
Build ingestion and pipelines using Lakeflow Connect, Auto Loader, Lakeflow pipelines, Spark, SQL and Delta Lake; at senior level, set patterns for other engineers.
Model data for consumption through data products, Databricks SQL and AI/BI dashboards, feature tables and model serving with MLflow; at senior level, decide the modelling approach with the client.
Migrate workloads from legacy warehouses, Hadoop or notebooks to Databricks, and refactor workloads that cannot be migrated as they are.
Use Git, CI/CD, Databricks Asset Bundles, Terraform, automated tests and code review; at senior level, run reviews and own the release process.
Run and optimise monitoring, job orchestration, performance and costs, including cluster sizing, serverless, Photon and query tuning.
Support governance where included in the engagement, covering data quality expectations in pipelines, lineage and access requests in Unity Catalog, together with the governance engineer.
At senior level, estimate and plan workstreams, support bids with the Databricks Lead, coach engineers and keep team certifications current.

Benefity

Yearly financial bonus.
Private medical care with Medicover, with additional packages such as dental, senior care and oncology available on preferential terms.
Life insurance and access to the NAIS benefit platform.
Access to over 70 training tracks with certification opportunities on the NEXT platform.
Free access to the Education First languages platform, TED Talks and Udemy Business materials and trainings.
Home office package including a laptop, monitor and chair.

Premie

Opieka zdrowotna

Ubezpieczenie

Szkolenia wewnętrzne

Kursy językowe

Capgemini

792 aktywne oferty

Zobacz wszystkie oferty

Aplikuj teraz