Scala/Spark Developer

Directio Sp. z o.o. is an AI-first software engineering company delivering innovative AI, data, cloud, and IoT solutions for clients across Europe and beyond.

• Strong commercial experience developing production applications in Scala, including functional programming concepts (immutability, pattern matching, higher-order functions), Scala type system, collections, sbt, and testing frameworks such as ScalaTest or munit.
• Proven hands-on experience with Apache Spark, including Spark SQL, DataFrame/Dataset API, ETL/ELT pipelines, performance optimization (partitioning, shuffle optimization, broadcast joins, caching), and troubleshooting using Spark UI.
• Experience working in production environments with large-scale data processing and distributed data pipelines.
• Strong understanding of software engineering best practices, code quality, performance optimization, and maintainability.

Nice to have:

• Experience with Kafka and Spark Structured Streaming.
• Experience with Airflow or similar workflow orchestration tools.
• Experience with cloud platforms such as AWS (EMR, S3, Glue), Azure, GCP (Dataproc), or Databricks.
• Knowledge of big data ecosystem technologies including Hadoop, Hive, Delta Lake, Parquet, and Avro.
• Experience with SQL, data modeling, data warehouses, and data lake architectures.
• Experience with Docker, Kubernetes, Git, and CI/CD pipelines.
• Familiarity with Scala libraries such as Cats, ZIO, Akka, or Pekko.
• Knowledge of Python or Java.

• Design, develop, and maintain scalable data processing applications using Scala and Apache Spark.
• Build, optimize, and maintain ETL/ELT pipelines processing large volumes of data.
• Develop efficient Spark applications using Spark SQL and the DataFrame/Dataset API.
• Optimize Spark jobs by improving partitioning strategies, shuffle operations, caching, and join performance.
• Troubleshoot and resolve performance issues using Spark UI and monitoring tools.
• Collaborate with data engineers, software developers, and stakeholders to design scalable data processing solutions.
• Participate in code reviews and contribute to software quality, performance, and engineering best practices.