Senior Data Engineer (Databricks)

Addepto is a leading AI consulting and data engineering company focused on Artificial Intelligence and Big Data. The company works with large enterprises and startups, including Rolls Royce, Continental, Porsche, ABB, and WGU. Addepto develops scalable AI solutions and has its own product called ContextClue, contributing open-source tools to the AI community. It is part of KMS Technology, offering enterprise-scale delivery capabilities.

• At least 5 years of commercial experience in Big Data system implementation, development, or maintenance.
• Strong programming skills in Python with clean code and OOP design.
• Proficient in SQL including performance tuning and experience with data warehousing.
• Experience designing and implementing data governance and management processes.
• Deep expertise in Big Data technologies including Apache Airflow, Dagster, Databricks, Spark, and DBT.
• Experience deploying solutions in cloud environments, preferably Azure.
• Knowledge of building and deploying Power BI reports and dashboards.
• Excellent understanding of dimensional data and data modeling techniques.
• Consulting experience guiding architectural decisions and technology selection.
• Ability to work independently and take ownership of project deliverables.
• Master's or Ph.D. in Computer Science, Data Science, Mathematics, Physics, or related field.

Nice to have:

• Experience with CI/CD, Kubernetes, Kafka, Dagster, and dbt.

• Design and optimize scalable data processing pipelines for streaming and batch workloads using Databricks, Apache Airflow, and Dagster.
• Architect and implement end-to-end data platforms ensuring availability, performance, and reliability.
• Lead development of CI/CD and MLOps processes for deployment automation and model lifecycle management.
• Develop and maintain applications for processing and analyzing diverse data sources.
• Collaborate with Data Science teams on ML projects including feature engineering and predictive model deployment.
• Design and manage complex data transformations using Databricks, DBT, and Apache Airflow.
• Translate business requirements into scalable technical solutions ensuring data quality.
• Ensure adherence to data security, compliance, and governance best practices across pipelines.