Principal Data Platform Engineer for Databricks

We are a global, collaborative team of engineers, data scientists, and product managers leveraging Generative AI to solve high-impact challenges. Our mission is to evolve Sightline, a premier Databricks-based Lakehouse platform.

Background & Leadership:

• 12+ years of professional experience in global IT projects, with a proven track record of managing senior stakeholders and business end-users.
• Software Engineering DNA: strong coding background rather than pure analytics or data science profile.
• Language Proficiency: B2+ or higher in written and spoken English.
• Comfortable shaping a vision and building a platform from the ground up in a fast-paced environment.

Technical Expertise:

• Deep experience with Databricks Lakehouse architecture, Structured Streaming, and Delta Live Tables.
• Practical knowledge of Azure Functions, Service Bus, Event Hub, App Services, and databases (Cosmos DB, MongoDB).
• Hands-on experience with CI/CD pipelines and managing cloud infrastructure via Terraform or similar IaC tools.
• Proficiency using modern AI agents and tools such as GitHub Copilot, Cursor, Claude Code, or Gemini CLI.
• Understanding of integration patterns between data platforms and consumption tools like Power BI.

Platform Evolution:

• Act as the Tech Lead for the Databricks Lakehouse, designing abstraction layers that simplify developer workflows and move beyond "batch-only" ETL thinking.

Streaming-First Strategy:

• Drive the transition from legacy warehouse paradigms to event-driven, near-real-time architectures using Delta Live Tables and Structured Streaming.

Engineering Leadership:

• Manage a small, specialized team of DevOps and Platform Engineers while mentoring cross-functional teams on best practices and toolsets.

Operational Excellence:

• Own the "DataOps" lifecycle, including CI/CD, Infrastructure as Code (IaC), and the optimization of Databricks consumption/costs.

Strategic Integration:

• Bridge the gap between the platform and downstream technologies like Microsoft Fabric, Power BI, and Azure ecosystem services.