Spark Data Lakehouse Developer

ITLT helps partner companies transform ambitious ideas into digital reality, focusing on challenges, technology curiosity, and agility. The role involves joining the Data Platform team at a stable financial sector institution as a key person responsible for designing and implementing data reconciliation solutions in the ODS layer (MongoDB) and building and scaling a modern Data Lakehouse environment.

• Minimum 4 years of practical experience in production environments using Apache Spark (PySpark, Spark-SQL, DataFrames, Structured Streaming).
• Experience working with MongoDB (read/write operations, document modeling, indexing).
• Practical knowledge of Data Lakehouse architecture, including managing versioned tables using Apache Iceberg (or readiness to migrate skills from Delta Lake/Hudi).
• Proficiency in Python and SQL.
• Competence in CI/CD environments (automation using Jenkins) and related tools (Jira, Confluence).
• Knowledge of monitoring, metrics, and logging tools (Prometheus, Grafana).

• Design and implement processes (using Spark-SQL / PySpark) to compare data in the ODS layer (MongoDB) with source systems.
• Build, version, and maintain data layers in Data Lakehouse architecture (raw → bronze → silver → gold) using technologies like Apache Iceberg, Delta Lake, or Hudi.
• Integrate the Spark environment with MongoDB using a dedicated connector, optimizing reads/writes, schema inference, and partitioning.
• Implement Data Quality rules, detect duplicates, gaps, anomalies, and generate quality alerts.
• Perform performance tuning of Spark processes (e.g., caching, broadcast joins, Adaptive Query Execution) and optimize cost of analytical queries.
• Automate deployments using Docker, Kubernetes, Spark Operator, Jenkins and develop and configure application monitoring with Prometheus and Grafana.
• Work in Agile methodology (Scrum/Kanban) and mentor Data Engineers and analysts.

Administrator of personal data is IT Leasing Team, based in Warsaw. Personal data is processed for recruitment purposes in accordance with legal requirements. Candidates have rights regarding their data, including access, correction, deletion, objection, and complaint rights. Providing data required by labor law is mandatory; other data is voluntary. Data processing duration aligns with recruitment procedures and consent. Data recipients include Just Join IT and other entities involved in recruitment.