Senior Data Engineer (k/m)

Lumicode is part of the Pentacomp Group, a Polish IT group with over 30 years of experience, delivering projects for large organizations and the public sector. Currently seeking a Data Warehouse Developer with Oracle and SAS experience for a long-term IT project in the insurance industry.

• Strong knowledge of Azure Databricks with experience in Python/PySpark, Spark SQL, and Delta Lake.
• Understanding of Bronze → Silver → Gold (Medallion Architecture).
• Experience integrating and processing data from API, JSON, and files.
• Practical knowledge of Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Storage.
• Experience with Databricks Workflows / Jobs; knowledge of Delta Live Tables / Lakeflow is a plus.
• Knowledge of schema enforcement and schema evolution.
• Implementation experience of data quality, validation, and reconciliation rules.
• Familiarity with data transformation and standardization processes.
• Experience in metadata management and data lineage.
• Knowledge of Unity Catalog and governance concepts.
• Experience designing API-based data ingestion and export processes.
• Work with metadata of documents and references to PDF/eLabel resources.
• Skills in performance optimization and data partitioning.
• Experience configuring CI/CD pipelines for notebooks and code with automated tests.
• Familiarity with monitoring, error handling, and reprocessing mechanisms.
• Experience with pipelines requiring traceability, auditability, reproducibility, and controlled releases.
• Experience in regulated environments with strict compliance and data quality requirements.
• English language skills sufficient for communication in an international environment.

Nice to have:

• Experience in pharmaceutical, medical, or other heavily regulated industries.
• Knowledge of eLabeling and regulatory content processes.
• Hands-on experience with Delta Live Tables / Lakeflow.
• Familiarity with metadata management and data lineage topics.

• Design, develop, and maintain data pipelines using Azure Databricks, Python, PySpark, Spark SQL, and Delta Lake.
• Implement solutions following the Bronze → Silver → Gold (Medallion Architecture).
• Extract and process data from API, JSON files, Azure Storage, and corporate systems.
• Build and maintain data quality mechanisms, validations, reconciliation, monitoring, and automated data tests.
• Implement data quality rules, schema validation, and operational controls.
• Optimize performance, partitioning, and scalability of data processing.
• Implement CI/CD processes, automated tests, error handling, and data reprocessing.
• Ensure traceability, auditability, reproducibility, and controlled releases of pipelines in a regulated environment.
• Collaborate with business and technical teams in designing and developing the data platform.

By submitting the recruitment form and CV, candidates consent to their personal data processing by Lumicode Sp. z o.o. for recruitment purposes, as detailed under GDPR regulations. Data will be stored until the expiry of claims related to the recruitment process and may be processed for future recruitment with consent. Candidates have rights to access, correct, delete, or limit processing of their data. No automated decision-making or profiling will occur. Contact details for data protection officer and withdrawal of consent are provided.