[GFA] Azure Senior Data Engineer

Software Mind develops technology solutions impacting global companies, working with tech giants and unicorns on transformative projects involving emerging technologies.

• Proficient in Databricks Lakehouse architecture and PySpark development
• Extensive experience in data modeling including fact and dimension tables, measures, grain analysis, SCD Type 2, surrogate keys, late-arriving records handling, overlap joins, and incremental loads with idempotency
• Advanced Python and PySpark skills for ETL/ELT development
• Deep knowledge of Databricks optimization features: optimize, zOrder, Liquid clustering, ACID transactions, performance tuning
• Proven experience in event-driven data architecture
• Experience with Azure data platform including datasets and data pipelines
• Strong SQL skills for complex transformations
• Experience handling large-scale complex datasets efficiently
• Experience with CI/CD automated deployment pipelines
• Understanding of basic networking concepts
• Familiarity with Scrum and agile methodologies

Nice to have:

• Understanding of stream processing and Spark Structured Streaming
• Experience with Infrastructure as Code tools such as Terraform, Bicep
• Experience with containerized applications: Azure Container Apps, Kubernetes
• Knowledge of Azure cloud-native solutions: Azure Data Factory, Azure Function App, Azure Container Instances

• Develop reusable, metadata-driven data pipelines using Databricks Lakehouse architecture and PySpark
• Design and implement comprehensive data models
• Build robust ETL/ELT solutions with advanced features such as Merge operations and SCD Type 2 implementations
• Implement incremental data loads with idempotency patterns and overlap joins optimization
• Automate and optimize data platform processes focusing on performance and reliability
• Build integrations with data sources and consumers using event-driven patterns
• Collaborate with infrastructure engineering team to set up cloud resources
• Initiate and implement improvements to data platform architecture

This role requires candidates to be based in Poland.