Senior Data Engineer – Scala / Spark / Hadoop (f/m)

B2Bnetwork is hiring a Senior Data Engineer to work on a strategic regulatory initiative within the financial sector. The role involves working in an international team developing and maintaining a data platform and reporting processes in a complex Big Data environment, including the use of technologies such as Scala, Hadoop, Spark, Kafka, Airflow, and Jenkins.

• Strong hands-on experience with Scala.
• Commercial experience with Apache Hadoop.
• Practical knowledge of Apache Spark and large-scale data processing.
• Experience building and maintaining Data Engineering pipelines.
• Experience with Apache Kafka and event-driven architecture.
• Practical knowledge of CI/CD processes, particularly Jenkins.
• Very good knowledge of Git, code review and pull request processes.
• Strong SQL development and optimisation skills.
• Experience supporting production environments.
• Ability to independently investigate complex technical issues.
• English proficiency sufficient for daily work in an international team.
• Availability to work from the Warsaw or Gdańsk office twice per week.
• Ability to perform the work from Poland.

Nice to have:

• Kubernetes experience.
• Knowledge of Apache Flink.
• Experience with Apache Airflow or Apache Oozie.
• Previous involvement in regulatory or reporting projects.
• Experience in the financial sector.

• Design, develop and maintain Data Engineering solutions using Hadoop and Apache Spark.
• Build scalable solutions for processing large volumes of data.
• Analyse complex business requirements and translate them into effective technical solutions.
• Develop and optimise SQL queries and data-processing workflows.
• Build and enhance data pipelines and reporting processes.
• Create and maintain workflows using Apache Oozie and Apache Airflow.
• Develop data solutions using Scala.
• Work with event-driven architecture based on Apache Kafka.
• Maintain and improve CI/CD processes using Git, pull requests and Jenkins.
• Provide L3 support for production environments.
• Investigate production incidents and technical problems in cooperation with relevant operational teams.
• Collaborate with business stakeholders to define requirements and analyse issues.
• Ensure the quality, performance, security and reliability of data solutions.

Availability to work from the Warsaw or Gdańsk office twice per week and ability to perform the work from Poland.