Data Engineer ( Spark / Streaming / Java)

KMD Poland is developing an innovative solution, KMD Elements, on Microsoft Azure cloud dedicated to the energy distribution market including electrical energy, gas, water, and utilities. The product supports institutions and companies in the energy market such as transmission service operators, market regulators, distribution service operators, energy trading, and retail companies. KMD Elements covers the full lifecycle of a customer on the energy market including meter data processing, network connection, physical network management, operator change, billing, payment, debt management, customer communication, and account termination. The solution supports complex billing models and scalability for large data volumes, promoting sustainable and green energy generation and consumption.

• 4+ years of Apache Spark experience with batch or streaming data engineering challenges
• Interest in stream processing with Apache Spark Structured Streaming on Apache Kafka
• Experience leading technical solution designs
• Experience with distributed systems on a cloud platform
• Experience with large-scale systems in a microservice architecture
• Familiarity with Git and CI/CD practices and ability to design or implement deployment processes for data pipelines
• Proactive approach and can-do attitude
• Excellent written and spoken English and Polish
• Higher education in computer science or related field
• Team player with strong communication skills

Nice to have:

• Apache Spark Structured Streaming
• Azure
• Domain Driven Development
• Docker containers and Kubernetes
• Message brokers (Kafka) and event-driven architecture
• Agile/Scrum

• Develop and maintain the leading IT solution for the energy market using Apache Spark, Databricks, Delta Lake, and Apache Kafka
• Have end-to-end responsibility for the full lifecycle of features developed
• Design technical solutions for business requirements from the product roadmap
• Maintain alignment with architectural principles defined on the project and organizational level
• Ensure optimal performance through continuous monitoring and code optimization
• Refactor existing code and enhance system architecture to improve maintainability and scalability
• Design and evolve the test automation strategy, including technology stack and solution architecture
• Prepare reviews, participate in retrospectives, estimate user stories, and refine features ensuring readiness for development