Data Engineer w/ Databricks

We are The Codest - International Tech Software Company with tech hubs in Poland delivering global IT solutions and projects. Our expertise centers on web development, cloud engineering, DevOps and quality. We have developed a product called Yieldbird, honored as a laureate of the Top25 Deloitte awards. Our mission is to help tech companies build impactful products and scale their IT teams through boosting IT delivery performance.

• 6+ years of experience in data engineering, designing and deploying production-grade data pipelines and ETL/ELT workflows
• Proficiency in Python, PySpark, and SQL
• Extensive hands-on experience with AWS services such as Glue, Lambda, Redshift, and S3
• Solid background in data governance and management tools including Unity Catalog and AWS Sagemaker Unified Studio
• Proven experience in data migration initiatives and modernizing legacy systems
• Communicative English language skills

Nice to have:

• Databricks Certified Professional Data Engineer or equivalent certification
• Excellent communication and teamwork skills
• Strong analytical thinking and problem-solving ability
• Experience with containerization and orchestration tools like Docker and Kubernetes

• Experience with Medallion Architecture (Bronze, Silver, Gold) for lakehouses using optimized performance patterns
• Build strong data quality frameworks with automated testing and monitoring
• Implement Delta Lake features such as time travel, vacuum operations, and Z-ordering
• Develop and maintain complex ETL/ELT pipelines processing large-scale datasets daily
• Design and implement CI/CD workflows for data pipelines using Databricks Asset Bundles or equivalent tools
• Create real-time and batch data processing solutions with Structured Streaming and Delta Live Tables
• Optimize Spark jobs for cost efficiency and performance, leveraging cluster auto-scaling and resource management
• Develop custom integrations with Databricks APIs and external systems
• Design scalable data architectures using Unity Catalog, Delta Lake, and Apache Spark
• Establish data mesh architectures with governance and lineage tracking