Senior Data Engineer - AWS / Databricks

160 - 190 PLN/ godz./ godz.B2BB2B (netto)

SeniorHybryda·WarszawaFull-time·B2B

#471432·Dodano 5 dni temu·0

Źródło: justjoin.it

Aplikuj teraz

Tech Stack / Keywords

Apache Spark Python SQL DatabricksDelta Lake

Podsumowanie oferty

Design, build, and optimise cloud-based data platforms for analytics, AI, and data products. The project focuses on AWS and Databricks, with data platform work using Apache Spark and Delta Lake.

Wymagania

Strong hands-on experience with Apache Spark, Delta Lake, and production-grade batch and streaming pipeline development.
Advanced programming skills in Python and SQL, with a solid grasp of data modelling, data quality, and governance principles.
Demonstrable hands-on experience with the Databricks Lakehouse Platform, ideally on AWS, using services such as S3, Glue, EMR, Redshift, Kinesis, and Lambda.
Deep technical expertise in at least one major cloud (AWS, GCP, or Azure/Fabric) and openness to working across clouds; candidates with strong Azure/Fabric experience are welcome if they want to expand their multi-cloud skills.
Solid understanding of distributed systems, lakehouse architectures, scalability, and system performance tuning.
A proactive, consultative mindset, including comfort asking probing questions, challenging assumptions, and collaborating in cross-functional agile teams.

Nice to have:

Hands-on experience across multiple clouds or direct experience in cloud-to-cloud migrations, for example Azure to GCP.
Practical experience with Google Cloud Platform services, including BigQuery, Dataflow, Pub/Sub, Cloud Storage, and Dataproc.
Familiarity with Snowflake.
Hands-on exposure to GenAI data pipelines, vector databases, and LLM data preparation.
Experience with Infrastructure as Code (Terraform, CloudFormation) and CI/CD pipelines for automated data deployments.
Knowledge of event streaming technologies such as Apache Kafka, or ML/experimentation tooling such as Databricks MLflow.

Obowiązki

Design, implement, and optimise scalable batch, streaming, and near-real-time data platforms using Apache Spark and Delta Lake, primarily on AWS and Databricks, with GCP or Azure/Fabric where required.
Build ingestion, transformation, and curation workflows for structured and unstructured data, applying lakehouse patterns and medallion layering (bronze, silver, gold).
Support cloud-to-cloud migrations, including re-platforming legacy pipelines, data models, and storage layers.
Deliver production-grade, model-ready datasets and build data pipelines for machine learning, causal modelling, and GenAI use cases, including LLMs, vector search, RAG, and agent-based architectures.
Design logical and physical data models for analytical and operational use cases, and orchestrate automated workflows with Databricks Workflows, Airflow, dbt, or cloud-native equivalents.
Enforce enterprise data governance, lineage, quality frameworks, and granular access controls (RBAC/ABAC) using Unity Catalog, AWS Lake Formation, or equivalent cloud tools.
Implement platform observability and monitor data freshness, pipeline reliability, cost efficiency, and SLA compliance.
Build serving layers (APIs, feature inputs, analytical endpoints) and partner with data scientists, ML engineers, and business stakeholders to challenge requirements and surface architectural solutions.

RemoDevs

20 aktywnych ofert

Zobacz wszystkie oferty

Aplikuj teraz