Senior Data Platform Engineer

180 - 200 PLN/ godz./ godz.B2BB2B (netto)

SeniorZdalnieFull-time·B2B

#478863·Dodano 3 dni temu·5

Źródło: nofluffjobs.com

Aplikuj teraz

Tech Stack / Keywords

Apache AirflowADFAzureKey VaultNetworkingStorageAzure DevOpsCI/CDInfrastructure as CodeDatabricks UNITYAzure Data FactoryAirflow TerraformData LakeRBACCI/CD PipelinesData engineeringPython SQL PySparkADLSSecurityCommunication skills

Podsumowanie oferty

Evolve the Azure-based Data Platform and Lakehouse architecture, owning platform foundations used by data product teams. The work covers Apache Airflow orchestration, Azure resource management, Terraform infrastructure as code, and Databricks governance and observability.

Wymagania

6+ years of experience in data engineering, including significant experience with platform or infrastructure topics.
Strong technical expertise in Python and SQL at an advanced level, including PySpark and Delta Lake semantics.
Deep hands-on production experience with Apache Airflow, including DAG design, deployment, operations, upgrades, and orchestration of Databricks and ADF workloads.
Hands-on experience managing Azure resources, including Key Vault, managed identities, service principals, secret management, Azure RBAC, Entra ID, and Unity Catalog permissions.
Experience with Azure DevOps repositories, pipelines, service connections, environments, and CI/CD for data platforms.
Knowledge of ADLS, networking and security basics, and Azure Data Factory.
Experience with Infrastructure as Code using Terraform and multi-environment promotion.
Hands-on experience with Databricks and Unity Catalog, including compute, jobs, and governance.
Experience with observability, monitoring, alerting, and incident management.
Solid understanding of Data Lakehouse architecture, Medallion architecture, and metadata-driven architecture; awareness of cost management and performance.

Obowiązki

Orchestration Platform (Apache Airflow & ADF)

Own the Apache Airflow platform, including environments, deployments, upgrades, configuration, connections, variables, pools, and DAG standards.
Migrate from Azure Data Factory to Airflow and evolve the metadata-driven orchestration framework, including DAG generation, reusable operators, dependencies, and scheduling patterns.
Ensure Airflow reliability and observability through SLAs, alerting, retries, backfills, health monitoring, and clear failure ownership.

Azure Resource Management

Manage Azure resources with Terraform, including Databricks workspaces, Data Lake Storage, Data Factory, Key Vault, networking, and supporting services.
Own secrets and identity management, including Key Vault, service principals, managed identities, secret scopes, and rotation.
Own access management, including Azure RBAC, Entra ID groups, and Unity Catalog grants and ownership; follow least privilege and manage access requests and periodic reviews.
Own Azure DevOps repositories, CI/CD pipelines, branch policies, multi-environment promotion, deployment validation, and Terraform state.
Track Azure resource costs and governance through tagging, budgets, right-sizing, and cost attribution.

Hands-on Data Platform evolution

Contribute to implementing the Data Catalog, Data Quality Framework, and lineage across the platform.
Drive observability so pipelines are observable, debuggable, and alert-driven, with clear ownership of failures.
Operate the Databricks platform, including Unity Catalog structure, cluster and job policies, Auto Loader ingestion patterns, and workload performance and cost optimization.

Technical Leadership

Participate in architecture reviews and technical decision-making, stating trade-offs involving performance, cost, reliability, and complexity.
Define and promote engineering standards, naming conventions, patterns, and guidelines through code and design reviews.
Collaborate with Architects, Product Owners, and data engineers on the technical maturation of data products.
Build and maintain developer tooling and automation, including AI-assisted engineering workflows such as shared agent instructions, skills, and automated PR review.

Support & Operations (Run Team Collaboration)

Provide second-line support to the Run/Operations team for complex production topics.
Troubleshoot and resolve platform incidents, failures, and performance issues; lead root-cause analysis and post-mortems.
Ensure proper handover, documentation, and operational readiness of platform changes.
Continuously improve runbooks, monitoring, and operational processes.

Benefity

Sport subscription.
Training budget.
Private healthcare.

Karta sportowa

Dofinansowanie szkoleń

Opieka zdrowotna

Margo

41 aktywnych ofert

Zobacz wszystkie oferty

Aplikuj teraz