Data Engineer / Senior Data Engineer

The role is for one of the largest financial institutions in Poland operating in the insurance industry, focusing on designing, developing, and optimizing data processing workflows within the Microsoft Fabric environment.

• Minimum 3 years of experience as a Data Engineer in production projects.
• Strong knowledge of Python including PySpark, and SQL for working with large data sets.
• Practical experience with at least one cloud data platform such as Microsoft Fabric, Azure Synapse, or Databricks.
• Knowledge of layered data architectures (e.g., medalion architecture) and file formats Delta and Parquet.
• Experience working collaboratively using Git.
• Ability to integrate diverse data sources including Excel files from business users.
• Willingness to sign an NDA and undergo security verification.

Nice to have:

• Certifications DP-600 or DP-700.
• Experience in insurance, finance, or highly audited environments.
• Knowledge of R language (used alongside Python).
• Practical experience with CI/CD for data platforms.
• Familiarity with data quality control frameworks.
• Work experience following Agile and Scrum methodologies.

• Designing and implementing pipelines using Data Factory and PySpark notebooks in Microsoft Fabric.
• Connecting file and database sources, including technical validation and format standardization.
• Transforming data and preparing batch data.
• Implementing data quality control rules and error reporting mechanisms.
• Versioning data sets and recording metadata such as parameters, sources, and versions.
• Following Git standards: storing pipeline code in repositories, deployments via CI/CD, and participating in code reviews.
• Collaborating with the Risk Team to define data transformation rules.
• Ensuring compliance with data privacy and security regulations.
• Integrating company security standards within the Azure environment.
• Optimizing processes for performance and cost efficiency.

Candidates must be willing to sign an NDA and undergo security verification.