Mid Data Engineer

CommIT operates in a financial domain handling millions of daily financial events including bets, wallet transactions, and live odds, servicing over 12 million active users. The role focuses on ownership of the data lakehouse architecture across AWS S3 and Snowflake/Databricks, supporting teams in Analytics, Finance, and Regulatory sectors.

• 3+ years hands-on in a production lakehouse environment.
• Lakehouse architecture — bronze/silver/gold layering, an open table format (Iceberg, Delta, or Hudi), schema evolution.
• Data layout & query optimization at TB+ scale — partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake.
• Cloud lakehouse/DWH in production — Snowflake, Databricks, or BigQuery.
• CDC & streaming ingestion — Kafka + Debezium or equivalent; late, duplicate, and out-of-order events.
• Strong SQL and data modeling skills.
• Correctness — freshness SLAs, drift detection, reconciliation against source wallet/ledger systems.
• Governance — catalogs, lineage, row/column access control, PII masking, retention, GDPR deletion.
• Cloud object storage — S3 or GCS, plus storage tiering and archival.
• Python and an orchestrator — Airflow or Dagster.

Nice to have:

• Experience in Fintech, iGaming, or regulated, audit-heavy environments.
• Cost monitoring / FinOps for storage and compute spend.
• Experience with Hudi and Dagster specifically.
• Knowledge of immutability / WORM regulatory retention.

• Own the lakehouse architecture: bronze/silver/gold layers, Iceberg/Delta tables, schema evolution.
• Land operational data via CDC streaming (Kafka, Debezium), handling late and duplicate events.
• Design data layout for speed and cost: partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake.
• Own retention and archival: storage tiering, regulatory retention, immutability, GDPR deletion.
• Guarantee correctness: freshness SLAs, drift detection, reconciliation against source wallet and ledger systems.
• Own governance: catalog and lineage, row/column access control, PII masking, encryption, audit trails.
• Monitor ingestion health, data anomalies, and cloud storage/compute spend.

Location requirement: Kraków, Poland. Hybrid work with 2 days per week from the office.