Lead Data Engineer (Spark)

Addepto is a leading AI consulting and data engineering company focusing on Artificial Intelligence and Big Data. They build scalable AI solutions for large enterprises and startups, including clients like Rolls Royce, Continental, Porsche, ABB, and WGU. The company has developed its own product, ContextClue, and contributes open-source solutions to the AI community. Addepto is part of KMS Technology, a US-based global technology group, combining AI specialization with enterprise-scale delivery capabilities.

• 7+ years of commercial experience in implementing, developing, or maintaining Big Data systems.
• Strong programming skills in Python or Java/Scala with clean code and OOP design.
• Experience designing and implementing data governance and data management processes.
• Familiarity with Big Data technologies including Spark, Cloudera, Kafka, Airflow, NiFi, Docker, Kubernetes, and Iceberg.
• Proven experience deploying solutions in cloud environments, preferably AWS.
• Excellent understanding of dimensional data and data modeling techniques.
• Excellent communication skills with client consulting experience.
• Ability to work independently and take ownership of project deliverables.
• Master's or Ph.D. in Computer Science, Data Science, Mathematics, Physics, or related field.
• Fluent in English at C1 level.

• Design and develop scalable data management architectures, infrastructure, and platform solutions for streaming and batch processing using Big Data technologies like Apache Spark, Hadoop, and Iceberg.
• Design and implement data management and data governance processes and best practices.
• Contribute to the development of CI/CD and MLOps processes.
• Develop applications to aggregate, process, and analyze data from diverse sources.
• Collaborate with the Data Science team on data analysis and Machine Learning projects, including text/image analysis and predictive model building.
• Develop and organize data transformations using DBT and Apache Airflow.
• Translate business requirements into technical solutions and ensure optimal performance and quality.