Senior Data Engineer

H2B Group is a consulting company helping clients automate business processes using AI and enables IT specialists to develop their careers in the B2B model. They offer projects in fintech, medtech, edutech, and industrial automation sectors. They support clients' development through fast and measurable results while providing IT specialists opportunities for career growth and valuable experience.

• Very good knowledge of Apache Spark and Lakehouse architecture.
• Advanced proficiency in Python and SQL.
• Experience with Microsoft Azure services: Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2).
• Practical experience with Delta Live Tables (DLT) and Unity Catalog.
• Knowledge of Azure Synapse Analytics or Azure Data Explorer.
• Experience building and maintaining CI/CD processes using Azure DevOps and Git.
• Experience in infrastructure automation, monitoring, and log auditing.
• Basic familiarity with reporting layer tools such as Power BI for data model sharing and consumption.

Language requirements: English at minimum C1 level for international collaboration.

• Design and maintain scalable Azure Lakehouse data solutions using Azure Databricks, Azure Synapse Analytics, and Data Lake.
• Implement efficient ETL/ELT data pipelines (batch and streaming) using Python, SQL, and Apache Spark.
• Modernize and migrate platforms including implementation and standardization of Unity Catalog, platform component upgrades, and transition towards Data Lakehouse architecture.
• Build and maintain Databricks Workflows, Delta Live Tables (DLT), and metadata-driven frameworks.
• Ensure data reliability, consistency, security, optimize storage structures, and perform production incident root cause analysis.
• Optimize performance and costs of Spark compute resources, SQL queries, and cloud infrastructure (FinOps).
• Collaborate on CI/CD pipelines in Azure DevOps, infrastructure automation (IaC), and system metrics monitoring.
• Work closely with data architects, analysts, and business teams; promote software engineering best practices within the data team.

The data controller is H2B Group Sp. z o.o. located at UL. Abrahama 1A, 80-337 Gdańsk. Candidates have rights regarding personal data including access, rectification, deletion, processing limitation, objection, data portability, and lodging complaints with supervisory authorities, with data processed for recruitment purposes in compliance with applicable laws.