(Senior) Databricks Engineer

Brak informacji o wynagrodzeniu

SeniorHybryda·WarszawaFull-time

#472685·Dodano 2 dni temu·2

Źródło: nofluffjobs.com

Aplikuj teraz

Tech Stack / Keywords

Data engineeringDatabricksData modellingCI/CDInfrastructure as CodeSpark PySpark Scala SQL PythonCloudAzure AWSStorageNetworkingGit Terraform UNITY

Podsumowanie oferty

You will build Databricks lakehouse platforms for clients, covering platform setup, ingestion, pipelines, data models and operations. The work includes building data products for analytics and machine learning using Spark, SQL and Delta Lake.

Wymagania

Engineer: 3+ years in data engineering, including at least one year of hands-on Databricks production experience. Senior Engineer: 5+ years, including at least three years on Databricks; hands-on work in at least three listed areas and a record of leading the technical side of a workstream.
Strong Spark (PySpark or Scala) and SQL skills, and experience with Python, Delta Lake, Lakeflow pipelines and jobs, and Databricks SQL.
Working knowledge of one cloud platform (Azure preferred; AWS or GCP), including storage, identity and networking as they apply to Databricks.
Experience with Git and CI/CD, and with Databricks Asset Bundles or Terraform; both are required at senior level.
Databricks Certified Data Engineer Associate certification held or in progress; Professional certification at senior level.
Basics of Unity Catalog, including catalogs, permissions and lineage.
Clear communication with client data teams and fluent English; at senior level, ability to present a technical design to client architects.

Nice to have:

Interest in data governance.
Experience in financial services, energy, pharma or the public sector.

Obowiązki

Set up and configure Databricks platforms, including workspaces, compute policies, Unity Catalog, networking and identity; at senior level, own platform design and its cost model.
Build ingestion and batch and streaming pipelines using Lakeflow Connect, Auto Loader, Spark, SQL and Delta Lake; define patterns for other engineers at senior level.
Model data products, Databricks SQL and AI/BI dashboards, feature tables and model serving with MLflow; decide the modelling approach with the client at senior level.
Migrate workloads from legacy warehouses, Hadoop or notebooks to Databricks, and refactor workloads that cannot be migrated as they are.
Use Git, Databricks Asset Bundles, Terraform, automated tests and code review; at senior level, run reviews and own the release process.
Monitor and optimise job orchestration, performance and costs, including cluster sizing, serverless, Photon and query tuning.
Support governance where included in the engagement, including data quality expectations in pipelines, lineage and access requests in Unity Catalog, working with the governance engineer.
At senior level, estimate and plan workstreams, support bids with the Databricks Lead, coach engineers and keep certifications current across the team.
Own pipelines, data products and parts of the platform within a workstream; at senior level, own technical design, review engineers’ work and work with client architects.
Work with the Databricks Lead on architecture, the Senior Databricks Engineer - Governance when governance is part of the engagement, and client data teams.

Benefity

Yearly financial bonus.
Private medical care with Medicover, with additional packages available on preferential terms.
Life insurance.
Access to the NAIS benefit platform.
Access to over 70 training tracks with certification opportunities through the NEXT platform.
Free access to the Education First languages platform, TED Talks and Udemy Business materials and trainings.

Premie

Opieka zdrowotna

Ubezpieczenie

Szkolenia wewnętrzne

Kursy językowe

Karta sportowa

Parking dla aut

Parking dla rowerów

Telefon

Napoje w biurze

Dofinansowanie szkoleń

Capgemini

789 aktywnych ofert

Zobacz wszystkie oferty

Aplikuj teraz