Data Engineer (Databricks)

Addepto is a leading AI consulting and data engineering company focusing exclusively on Artificial Intelligence and Big Data. They build scalable, ROI-focused AI solutions for global enterprises and startups including Rolls Royce, Continental, Porsche, ABB, and WGU. Addepto has developed its own AI product, ContextClue, and contributes open-source solutions to the AI community. The company is part of KMS Technology, providing enterprise-scale delivery capabilities and AI specialization to move clients from experimentation to production at scale.

• At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems.
• Strong programming skills in Python, including clean code practices and OOP design.
• Strong SQL skills with experience in performance tuning, query optimization, and data warehousing solutions.
• Experience designing and implementing data governance and data management processes.
• Deep expertise with Big Data technologies including Databricks, Spark, and Apache Airflow.
• Experience in implementing and deploying cloud solutions, preferably on Azure.
• Knowledge of building and deploying Power BI reports and dashboards.
• Excellent understanding of dimensional data and data modeling techniques.
• Consulting experience to guide clients on architectural decisions and technology selection.
• Ability to work independently and take project ownership.
• Master’s or Ph.D. degree in Computer Science, Data Science, Mathematics, Physics or related field.

Nice to have:

• Experience with CI/CD, Kubernetes, Kafka, Airflow, Dagster, dbt.

• Design and optimize scalable data processing pipelines for streaming and batch workloads using Big Data technologies like Databricks, Apache Airflow, and Dagster.
• Architect and implement end-to-end data platforms ensuring availability, performance, and reliability.
• Lead development of CI/CD and MLOps processes for deployment automation, monitoring, and model lifecycle management.
• Develop and maintain applications for aggregating, processing, and analyzing diverse data sources efficiently and at scale.
• Collaborate with Data Science teams on Machine Learning tasks including feature engineering and predictive model deployment.
• Design and manage complex data transformations using Databricks, dbt, and Apache Airflow ensuring data integrity and consistency.
• Translate business requirements into scalable technical solutions optimizing performance and data quality.
• Ensure data security, compliance, and governance best practices across data pipelines.

Projects:

• Design and development of a universal data platform for global aerospace companies powered by Azure and Databricks.
• Data platform transformation for energy management association with comprehensive data catalog and data governance leveraging Databricks and Apache Spark/PySpark.
• Design of data transformation and data ops pipelines for a global car manufacturer handling real-time streaming and batch data, optimizing data model performance.