Senior Data Engineer

hubQuest is a team of tech enthusiasts building cutting-edge IT and Analytical Hubs to empower partners to become data-driven organizations. The role is within a client's Data Management & Data Foundation area focusing on strategic and technically challenging initiatives within a wider organization of approximately 30 engineers and technology specialists.

• 8+ years of professional experience in Data Engineering, Software Engineering, or related field.
• Strong hands-on commercial experience with Azure Databricks.
• Very good knowledge of the Azure cloud ecosystem and cloud-based data architectures.
• Excellent programming skills in Python.
• Advanced PySpark skills and experience with large-scale data processing workloads.
• Advanced SQL skills including complex transformations, optimization, and analytical/window functions.
• Strong understanding of data engineering principles, ETL/ELT patterns, data pipelines, distributed processing, and production platforms.
• Experience designing and delivering solutions independently.
• Ability to understand unfamiliar technical problems and drive implementation.
• Strong troubleshooting and problem-solving skills.
• Experience with production-grade engineering practices, including version control, testing, CI/CD, monitoring, and deployment.
• Ability to produce clean, maintainable, well-structured code.
• High ownership, independence, and proactivity.
• Excellent English communication skills (written and spoken).

Particularly valuable:

• Hands-on experience with AI capabilities within Databricks and AI-enabled data solutions.
• Experience with RAG architectures, LLM-based applications, or conversational enterprise data interfaces.
• Experience with Databricks Genie or similar conversational data discovery.
• Strong knowledge of Unity Catalog.
• Experience migrating enterprise data platforms to Databricks.
• Understanding of data cataloguing, metadata management, lineage, governance, and data discovery.
• Awareness of solution and data architecture to contribute to architectural decisions.
• Experience designing reusable frameworks and engineering standards.
• Performance optimization experience in Databricks / Spark environments.
• Experience with Azure DevOps and mature CI/CD practices for data solutions.
• Architectural experience is not required but advantageous.

• Design, develop, and optimize complex data solutions and data pipelines on Azure and Databricks.
• Take end-to-end technical ownership from problem understanding through solution design, implementation, testing, and production deployment.
• Write high-quality, production-grade code using Python, PySpark, and SQL.
• Work extensively with Azure Databricks and modern Databricks ecosystem capabilities.
• Explore and implement AI-enabled capabilities within the data platform, including Databricks AI features and RAG-based approaches.
• Contribute to migration and modernization of data platforms into Databricks architectures.
• Work with Unity Catalog for scalable data discovery, governance, and access.
• Solve complex or non-standard technical problems without existing blueprints.
• Identify opportunities to improve architecture, performance, scalability, reliability, and maintainability.
• Challenge existing approaches and contribute ideas.
• Collaborate with engineers, architects, product owners, DataOps, QA, DevOps, and other teams.
• Support architectural discussions and contribute to technical design decisions.
• Move between strategic initiatives as priorities evolve bringing strong expertise where needed.

The Controller of personal data for recruitment purposes is HUBQUEST spółka z ograniczoną odpowiedzialnością in Warsaw. Personal data is processed based on given consent in compliance with GDPR. Candidates have the right to access, rectify, or request erasure of personal data, which equates to resigning from recruitment. Data will not be transferred outside the EU and will not be subject to automated processing or profiling. Participation in recruitment requires providing personal data. Contact with Data Protection Officer is available.