Data Engineer

This position is part of a complex and dynamic migration programme in the pharmaceutical industry, pivotal for ensuring the successful transition of data systems.

• Senior hands-on experience with Azure Databricks (jobs, clusters, Unity Catalog, SQL warehouses).
• Expertise in PySpark / Python for transformation logic and ETL pipelines.
• Solid working knowledge of Delta Lake (schema evolution, MERGE, time travel).
• Advanced SQL skills (complex queries, window functions); ability to comprehend legacy ETL.
• Familiarity with Databricks Asset Bundles (DAB) and CI/CD practices (Azure DevOps, Git).
• Working knowledge of Azure Data Lake Storage Gen2.
• Comfortable understanding legacy ETL processes and source data profiling.
• Experience with data reconciliation (defining criteria, interpreting discrepancies).
• Ability to articulate requirements for junior engineers.
• Fundamental knowledge of Power BI / Tableau.

Nice to have:

• Experience in platform engineering and configuration refactoring using YAML.
• Ability to maintain environment governance across DEV, QAS, and PRD.
• Skills in source feasibility analysis for new data sources.
• Capability in transformation development across various data zones using PySpark and Delta Lake.
• Familiar with implementing acceptance criteria and conducting quality checks.

• Develop and maintain data pipelines using Azure Databricks.
• Implement ETL processes, ensuring data quality and integrity.
• Read and comprehend complex SQL stored procedures.
• Translate architecture direction into engineering delivery.
• Conduct data profiling and quality assessments.
• Define acceptance criteria for data transformation.
• Collaborate with the programme team to support overall migration efforts.
• Participate in CI/CD processes for efficient deployment.

Remote engagement options are available.