Data Engineer (Spark)

Addepto is a leading AI consulting and data engineering company that builds scalable, ROI-focused AI solutions for some of the world's largest enterprises and pioneering startups. The company specializes exclusively in Artificial Intelligence and Big Data. It has developed its own product called ContextClue and contributes open-source solutions to the AI community. Addepto is part of KMS Technology, a US-based global technology group, which combines AI specialization with enterprise-scale delivery capabilities.

• At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems, data governance, and data management.
• Strong programming skills in Python (or Java/Scala) with clean code and OOP design.
• Hands-on experience with Big Data technologies like Spark, Cloudera, Data Platform, Airflow, Kafka, NiFi, Docker, and Iceberg.
• Excellent understanding of dimensional data and data modeling techniques.
• Experience deploying solutions in cloud environments.
• Consulting experience with strong communication and client management skills.
• Ability to work independently and take ownership of project deliverables.
• Fluent in English (at least C1 level).
• Bachelor’s degree in technical or mathematical studies.

Nice to have:

• Experience with MLOps frameworks such as Kubeflow or MLflow.
• Familiarity with Databricks and/or dbt.

• Develop and maintain a high-performance data processing platform for automotive data, ensuring scalability and reliability.
• Design and implement data pipelines processing large data volumes in streaming and batch modes.
• Optimize data workflows for efficient ingestion, processing, and storage with technologies like Spark, Cloudera, and Airflow.
• Work with data lake technologies such as Iceberg to manage structured and unstructured data.
• Collaborate with cross-functional teams to integrate diverse data sources.
• Monitor and troubleshoot the platform to ensure high availability, performance, and data accuracy.
• Leverage cloud services including AWS for infrastructure management and scaling.
• Write and maintain high-quality code in Python (or Java/Scala) for data processing and automation tasks.