Software Architect - Databricks

At Square One Poland we link IT experts with the business. With over 25 years of experience, we specialize in recruitment processes on a global scale. Despite years of experience, we still have a startup DNA and this is our advantage. Our offices are located in London and Warsaw, however, we can reach clients from all over the world, from start-ups to big worldwide corporations.

• Cloud: Azure Databricks or AWS Databricks with native identity integration (Azure AD/AWS IAM).
• Databricks platform: Runtime ML, Mosaic AI, Model Serving, MLflow, Unity Catalog, Delta Lake, Feature Store, AutoML, Databricks Workflows, Databricks SQL.
• Data processing: Structured Streaming for real-time pipelines and Delta Lake for ACID storage, time travel, and reliability.
• Languages and APIs: Python with Databricks-native libraries, Databricks REST APIs and SDK.
• Frontend: React-based dashboard integrated with Databricks SQL for monitoring and human-in-the-loop controls.
• Security and compliance: Encryption in transit/at rest, enterprise governance with Unity Catalog, and SOC 2/GDPR aligned controls.
• Monitoring and operations: Native Databricks monitoring, custom alerting, performance dashboards, and A/B testing frameworks.
• Deployment and MLOps: Auto-scaling model endpoints, MLflow Model Registry, automated retraining pipelines, and blue-green deployments.

• Lead the end-to-end architecture of a Databricks-centric multi-agent processing engine leveraging Mosaic AI, Model Serving, MLflow, Unity Catalog, Delta Lake, and Feature Store for decoder automation at scale.
• Design governed data ingestion, storage, and real-time processing using Delta Lake, Structured Streaming, and Databricks Workflows with enterprise security and full lineage.
• Own the model lifecycle with MLflow: experiment tracking, registry/versioning, A/B testing, drift monitoring, and automated retraining pipelines.
• Architect low-latency model serving endpoints with auto-scaling and confidence-based routing for sub-second agent decisioning.
• Establish robust data governance with Unity Catalog across all environments.
• Drive performance and cost optimization
• Define production release strategies (blue-green), monitoring and alerting, operational runbooks, and SLOs for dependable operations.
• Partner with engineering, MLOps, and product to deliver human-in-the-loop workflows and dashboards using Databricks SQL and a React frontend.
• Lead change management, training, and knowledge transfer while managing a parallel shadow-processing path during ramp-up.
• Plan and coordinate phased delivery, success metrics, and risk mitigation across foundation, agent development, automation, and production rollout.