Senior Data Engineer (Spark)

• At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems and data governance and management processes.
• Strong programming skills in Python, Java, or Scala, including clean code and OOP design.
• Hands-on experience with Spark, Cloudera, Data Platform, Airflow, Kafka, NiFi, Docker, and Iceberg.
• Excellent understanding of dimensional data and data modeling techniques.
• Experience implementing and deploying solutions in cloud environments.
• Consulting experience, including direct client interaction, and excellent communication and client management skills.
• Ability to work independently and take ownership of project deliverables.
• Fluent English (at least C1 level).
• Bachelor’s degree in technical or mathematical studies.

Nice to have:

• Experience with an MLOps framework such as Kubeflow or MLflow.
• Familiarity with Databricks and/or dbt.

Your main responsibilities:

• Develop and maintain a high-performance data processing platform for automotive data, ensuring scalability and reliability.
• Design and implement data pipelines that process large volumes of data in streaming and batch modes.
• Optimize data workflows for efficient ingestion, processing, and storage using Spark, Cloudera, and Airflow.
• Work with data lake technologies such as Iceberg to manage structured and unstructured data.
• Collaborate with cross-functional teams to understand data requirements and integrate data sources.
• Monitor and troubleshoot the platform to ensure availability, performance, and data processing accuracy.
• Use AWS cloud services to manage infrastructure and scale processing workloads.
• Write and maintain high-quality Python, Java, or Scala code for data processing and automation.