Senior Data Engineer

DS STREAM is expanding its AI Data Engineering Team, focusing on a mature, high-availability data hub built within Azure Databricks using Unity Catalog, Spark Structured Streaming, and Asset Bundles. The project employs the Medallion Architecture framework to provide a scalable foundation for ingesting, transforming, and integrating new data sources into the existing ecosystem.

• At least 5 years of commercial experience in Data Engineering with a focus on large-scale data pipelines
• Proven experience with advanced solutions in Azure cloud and Databricks
• Previous exposure to GCP infrastructure and tools such as Airflow, Pub/Sub, BigQuery, VertexAI
• Knowledge of Spark Structured Streaming
• Proficiency in Python and SQL
• Ability to build pipelines using GitHub or similar tools

Nice to have:

• Ability to build enterprise API integrations with FastAPI or similar
• Experience in leading or mentoring engineers

• Develop programs for processing large volumes of data
• Ingest data from APIs
• Develop Databricks pipelines using PySpark and SQL
• Handle Spark Structured Streaming pipelines
• Design and manage objects in Unity Catalog according to Medallion Architecture
• Use Agile and Scrum methodologies

This is a 100% remote role within Poland only.