Data Engineer (Spark)

Addepto is a leading AI consulting and data engineering company that builds scalable, ROI-focused AI solutions for large enterprises and startups worldwide, including clients like Rolls Royce, Porsche, and ABB. The company specializes in Artificial Intelligence and Big Data and has developed its own product, ContextClue, while contributing open-source solutions to the AI community. Addepto is part of KMS Technology, a US-based global technology group, combining AI specialization with enterprise-scale delivery.

• At least 3 years of commercial experience in Big Data systems development, governance, and management.
• Strong programming skills in Python or Java/Scala, including clean code and OOP design.
• Hands-on experience with Spark, Cloudera, Airflow, Kafka, NiFi, Docker, and Iceberg.
• Excellent understanding of dimensional data and data modeling techniques.
• Experience implementing and deploying solutions in cloud environments.
• Consulting experience with excellent communication and client management skills.
• Ability to work independently and own project deliverables.
• Fluent in English at least at C1 level.
• Bachelor's degree in technical or mathematical studies.

• Develop and maintain a high-performance data processing platform for automotive data with scalability and reliability.
• Design and implement data pipelines that handle large volumes of data in streaming and batch modes.
• Optimize data workflows for efficient ingestion, processing, and storage using Spark, Cloudera, and Airflow.
• Manage structured and unstructured data using data lake technologies like Iceberg.
• Collaborate with teams to understand data requirements and integrate diverse data sources.
• Monitor and troubleshoot platform performance and data accuracy.
• Leverage cloud services (AWS) for infrastructure management and scaling workloads.
• Write and maintain high-quality Python (or Java/Scala) code for automation and processing.

Nice to have:

• Experience with MLOps frameworks such as Kubeflow or MLflow.
• Familiarity with Databricks and/or dbt.