Data Engineer - Azure Data Marketplace

Our client is a global professional services organisation delivering technology-enabled solutions across consulting, advisory and assurance services. They are building an advanced AI-powered data marketplace platform designed to transform how data assets are created, managed and consumed across the enterprise. The organisation invests heavily in innovation and technology, creating opportunities for experienced engineers to work on meaningful projects that impact business operations globally.

• 8-10 years of experience in data engineering roles
• Expert knowledge of Azure Data Factory pipeline development and management
• Proficiency with Azure Databricks, notebooks, jobs and clusters
• Strong programming skills in Python, PySpark and SQL
• Experience with Apache Spark for big data processing
• Knowledge of ETL processes and data transformation techniques
• Familiarity with Azure storage solutions including SQL Database, Blob Storage and Data Lake Storage
• Understanding of data warehousing principles and scalable data modelling
• Experience with PL/SQL
• Ability to write modular, testable and maintainable code
• Strong collaboration and communication skills
• Experience working in cross-functional team environments

• Identify and develop opportunities to improve data assets and services
• Design and build data ingestion pipelines focused on stability and optimisation
• Develop, test and maintain robust data pipelines and processes
• Write resilient, fault-tolerant and modular code for high availability
• Research and implement new methods to build world-class data products
• Collaborate across Ingestion, Consumption and Activation teams
• Manage Azure Data Factory pipelines, activities, triggers and data flows
• Process large datasets using Azure Databricks and Apache Spark
• Transform and integrate data from multiple sources
• Ensure data pipeline traceability and monitoring