Data Engineer

Brak informacji o wynagrodzeniu
SeniorFull-time
#354795·Dodano wczoraj·0
Źródło: emagine
Aplikuj teraz

Tech Stack / Keywords

Azure DatabricksETLSQLArchitectureProfilingCI/CDUnityPySpark

Firma i stanowisko

This position is part of a complex and dynamic migration programme in the pharmaceutical industry, pivotal for ensuring the successful transition of data systems.


Wymagania

  • Senior hands-on experience with Azure Databricks (jobs, clusters, Unity Catalog, SQL warehouses).
  • Expertise in PySpark / Python for transformation logic and ETL pipelines.
  • Solid working knowledge of Delta Lake (schema evolution, MERGE, time travel).
  • Advanced SQL skills (complex queries, window functions); ability to comprehend legacy ETL.
  • Familiarity with Databricks Asset Bundles (DAB) and CI/CD practices (Azure DevOps, Git).
  • Working knowledge of Azure Data Lake Storage Gen2.
  • Comfortable understanding legacy ETL processes and source data profiling.
  • Experience with data reconciliation (defining criteria, interpreting discrepancies).
  • Ability to articulate requirements for junior engineers.
  • Fundamental knowledge of Power BI / Tableau.

Nice to have:

  • Experience in platform engineering and configuration refactoring using YAML.
  • Ability to maintain environment governance across DEV, QAS, and PRD.
  • Skills in source feasibility analysis for new data sources.
  • Capability in transformation development across various data zones using PySpark and Delta Lake.
  • Familiar with implementing acceptance criteria and conducting quality checks.

Obowiązki

  • Develop and maintain data pipelines using Azure Databricks.
  • Implement ETL processes, ensuring data quality and integrity.
  • Read and comprehend complex SQL stored procedures.
  • Translate architecture direction into engineering delivery.
  • Conduct data profiling and quality assessments.
  • Define acceptance criteria for data transformation.
  • Collaborate with the programme team to support overall migration efforts.
  • Participate in CI/CD processes for efficient deployment.

Inne informacje

Remote engagement options are available.

emagine

emagine

197 aktywnych ofert

Zobacz wszystkie oferty
Aplikuj teraz