Senior Data Engineer – Spark / Data Lakehouse

Clurgo is a company created by developers for developers. Its teams encompass full competencies from software development and infrastructure, through analysis and testing, to strategic product and process management. They implement diverse IT projects for clients from various industries, focusing on good programming practices and work-life balance. The role is within the Data Platform team of a banking sector client.

• Minimum 4 years of practical experience with Apache Spark in production environments
• Experience with PySpark, Spark-SQL, DataFrames, and Structured Streaming
• Practical experience reading and writing data to/from MongoDB using MongoDB Spark Connector
• Experience with Data Lakehouse architecture managing versioned tables using Iceberg, Delta Lake, or Hudi
• Knowledge of Jenkins for automating unit/integration tests and deployment processes for Spark applications
• Familiarity with monitoring and observability tools like Prometheus and Grafana
• Fluent English communication skills (minimum B2)
• Experience working effectively in Agile methodologies (Scrum/Kanban)

Nice to have:

• Practical knowledge of Dynatrace and OpenTelemetry
• Experience with Docker and Spark Operator in Kubernetes environments

• Design and implement Spark-SQL/PySpark tasks to verify data consistency between ODS (MongoDB) and source systems
• Integrate Spark with MongoDB using MongoDB Spark Connector
• Optimize reads from MongoDB
• Build medalion architecture using Iceberg, Delta Lake, or Hudi
• Implement CI/CD for table schemas, migrations, and versioning
• Cost optimization of queries in the Data Lakehouse environment
• Configure metrics (Prometheus, Grafana), logs, and tracing (OpenTelemetry, Dynatrace) for Spark and batch/stream processes
• Define SLA/SLO and alerts
• Diagnose issues with records, delays, and data inconsistencies
• Collaborate with Data Engineers, Business Analysts, testers, and DevOps teams

The data controller is Clurgo sp. z o.o., Warsaw. Personal data processing is for recruitment purposes with rights to access, rectify, delete, and object. Providing mandatory data per labor code is obligatory; refusal may hinder recruitment. Consent can be withdrawn anytime.