Senior Data Platform Engineer

The authorized Informatica Corp, Altair and KMS Lighthouse distributor in Poland and CEE providing licenses, implementation services, trainings, onsite experts, and professional services.

• Minimum 5 years of experience in cloud environments as a data engineer, programmer, or related roles
• Strong knowledge of Python and ANSI SQL
• Practical experience with Azure Databricks Lakehouse technology
• Use of Databricks connectors for integration with source systems (REST API, JDBC, SaaS, files)
• Experience with stream processing technologies: Structured Streaming, Auto Loader
• Designing and orchestrating data pipelines (ETL/ELT) in medal architecture using Databricks and/or Azure Data Factory
• Knowledge of data governance and security: Unity Catalog, RBAC, access management
• Optimization in Spark: query tuning, file management, liquid clustering, Z-ORDER, OPTIMIZE/VACUUM
• Familiarity with Spark Declarative Pipelines: dependency definition, DAG management, integration with data pipelines
• Knowledge of Declarative Automation Bundles or similar deployment management approaches
• Usage of LLM/AI models for code generation in data transformation and extraction in Databricks
• English language proficiency enabling business meetings and documentation at least at B2 level
• Experience with project documentation
• Accuracy and proactive attitude in task execution, proposing solutions and scenarios

Nice to have:

• Experience with Structured Streaming, Auto Loader, ETL, ELT, Unity Catalog, RBAC, Spark tuning, and Spark liquid clustering

• Design and implement new solutions within a large data platform based on Azure Databricks technology
• Create and optimize SQL and Python code for efficient processing in a cloud data platform environment
• Build data flows following the current framework and both functional and non-functional requirements
• Develop and implement processes and best practices for data management and supervision
• Optimize and improve system configuration and data structures according to industry best practices
• Collaborate with system administrators to identify and design optimal interfaces for communication with the cloud data platform
• Work effectively with a team of data engineers and closely with analysts and a data platform architect