Capgemini

Senior Databricks Engineer - Governance

Brak informacji o wynagrodzeniu

SeniorHybryda·WarszawaFull-time

#463351·Dodano 14 dni temu·0

Źródło: Capgemini

Aplikuj teraz

Tech Stack / Keywords

DatabricksUnity CatalogSpark SQL PythonDelta LakeTerraformDatabricks Asset BundlesMLflowLakeflow

Podsumowanie oferty

Own the governance layer of client platforms, working on pipelines, data models and platform setup for Databricks engagements. Implement governance through Unity Catalog, including access controls, lineage, data quality and governance of AI assets.

Wymagania

5+ years of hands-on Databricks engineering in production, including at least two years designing and administering Unity Catalog at enterprise scale.
Deep knowledge of Unity Catalog, including its privilege model, row and column security, tags and attribute-based policies, system tables, lineage, Delta Sharing, Lakehouse Federation and migration from Hive metastore.
Strong Spark, SQL and Python skills; experience with Delta Lake, Lakeflow pipelines and jobs, infrastructure as code using Terraform and Databricks Asset Bundles, and CI/CD.
Working knowledge of cloud security on Azure or AWS as it applies to Databricks, including identity, storage access, private networking and key management.
Databricks Certified Data Engineer Professional or Associate certification.
Interest in data governance and regulation.
Ability to explain an access model or lineage graph to a non-technical data owner.
Fluent English.

Nice to have:

Platform Administrator accreditation.
Experience with a governance or catalog tool such as Purview or Collibra, or with GDPR, the EU AI Act or sector-regulated data.
Experience in financial services, energy, pharma or the public sector.

Obowiązki

Design and implement Unity Catalog for clients, including metastore and catalog structure, workspace binding, external locations and storage credentials, identity federation (SCIM, service principals) and ownership models.
Build, test and document access controls that reflect governance designs, including privileges, groups, row filters, column masks, tag-based attribute controls and data classification.
Produce governance evidence from platform system tables, including lineage, audit logs and billing; create access reviews and dashboards, and integrate with Microsoft Purview or Collibra where used by the client.
Implement data quality in pipelines using expectations in Lakeflow pipelines and Lakehouse Monitoring, and publish quality metrics to the catalog and governance team.
Govern AI assets on the platform with the AI governance team, including models and functions registered in Unity Catalog, MLflow approval workflows, AI Gateway policies for LLM endpoints and traceability logging.
Implement sovereignty and residency controls with the sovereignty team, including region placement, customer-managed keys, private connectivity, Delta Sharing and Lakehouse Federation.
Migrate clients from Hive metastore and legacy permission models to Unity Catalog.
Build reusable assets, including Unity Catalog reference setups as code using Terraform and Databricks Asset Bundles, access policy patterns and evidence dashboards; contribute to the governance-by-design reference architecture.

Benefity

Yearly financial bonus.
Private medical care with Medicover, with additional packages available on preferential terms.
Life insurance.
Access to the NAIS benefit platform.
Access to over 70 training tracks with certification opportunities on the NEXT platform.
Free access to the Education First language platform, TED Talks and Udemy Business materials and trainings.
Home office package including a laptop, monitor and chair.

Premie

Opieka zdrowotna

Ubezpieczenie

Kursy językowe

Capgemini

780 aktywnych ofert

Zobacz wszystkie oferty

Aplikuj teraz