Senior Data Engineer

hubQuest is a company focused on building cutting-edge IT and Analytical Hubs to empower partners to become data-driven organizations. The role is within the Data Management & Data Foundation area, part of a wider Data Management organization with approximately 30 engineers and technology specialists, working on strategic, technically challenging initiatives.

• 8+ years professional experience in Data Engineering, Software Engineering, or related field.
• Strong hands-on commercial experience with Azure Databricks.
• Very good knowledge of the Azure cloud ecosystem and cloud-based data architectures.
• Excellent programming skills in Python.
• Advanced PySpark skills with large-scale data processing experience.
• Advanced SQL including complex transformations, optimization, and analytical/window functions.
• Strong understanding of data engineering principles, ETL/ELT patterns, data pipelines, distributed processing, and production data platforms.
• Experience designing and delivering solutions independently.
• Ability to analyze unfamiliar technical problems and drive implementation.
• Strong troubleshooting and problem-solving skills.
• Experience with production-grade engineering practices: version control, testing, CI/CD, monitoring, deployment.
• Strong software development principles producing clean, maintainable, well-structured code.
• High ownership, independence, and proactivity.
• Excellent English communication skills.

Nice to have:

• Hands-on experience with AI capabilities within Databricks and AI-enabled data solutions.
• Experience with RAG architectures, LLM-based applications, or conversational interfaces.
• Experience with Databricks Genie or similar conversational data discovery.
• Strong knowledge of Unity Catalog.
• Experience migrating enterprise data platforms to Databricks.
• Understanding of data cataloguing, metadata management, lineage, governance, and data discovery.
• Strong awareness of solution and data architecture.
• Experience designing reusable frameworks, components, or engineering standards.
• Performance optimization experience in Databricks/Spark environments.
• Experience with Azure DevOps and mature CI/CD practices for data solutions.

• Design, develop, and optimize complex data solutions and data pipelines on Azure and Databricks.
• Take end-to-end technical ownership of assigned engineering problems from challenge understanding through production deployment.
• Write high-quality, production-grade code using Python, PySpark, and SQL.
• Work extensively with Azure Databricks and modern capabilities of the Databricks ecosystem.
• Explore and implement AI-enabled capabilities within the data platform including Databricks AI, RAG approaches, data discovery, and conversational access.
• Contribute to the migration and modernization of existing data platforms into Databricks-based architectures.
• Work with Unity Catalog and contribute to scalable data discovery, governance, and access approaches.
• Solve technically difficult or non-standard problems without existing blueprints.
• Proactively identify improvements in architecture, performance, scalability, reliability, and maintainability.
• Challenge existing approaches and actively contribute new ideas.
• Collaborate with Data Engineers, Architects, Product Owners, DataOps, QA, DevOps, and other teams.
• Support architectural discussions and contribute to technical design decisions.
• Move between strategic initiatives as priorities evolve, applying strong engineering expertise.