Data Engineer (Databricks)

Addepto is a leading AI consulting and data engineering company focused on building scalable, ROI-focused AI solutions for major enterprises and startups including Rolls Royce, Continental, Porsche, ABB, and WGU. The company specializes exclusively in Artificial Intelligence and Big Data and contributes open-source solutions to the AI community. It is part of KMS Technology, a global technology group, combining AI specialization with enterprise-scale delivery capabilities.

• At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems.
• Strong programming skills in Python with clean code and OOP design.
• Strong SQL skills including performance tuning, query optimization, and experience with data warehousing solutions.
• Experience designing and implementing data governance and data management processes.
• Deep expertise in Big Data technologies including Databricks, Spark, and Apache Airflow.
• Experience implementing and deploying solutions in cloud environments, preferably Azure.
• Knowledge of building and deploying Power BI reports and dashboards for data visualization.
• Excellent understanding of dimensional data and data modeling techniques.
• Consulting experience with ability to guide clients on architectural decisions, technology selection, and best practices.
• Ability to work independently and take ownership of project deliverables.
• Master's or Ph.D. in Computer Science, Data Science, Mathematics, Physics, or related field.

Nice to have:

• Experience with CI/CD, Kubernetes, Kafka, Airflow, Dagster, and dbt.

• Design and optimize scalable data processing pipelines for both streaming and batch workloads using Big Data technologies such as Databricks, Apache Airflow, and Dagster.
• Architect and implement end-to-end data platforms, ensuring high availability, performance, and reliability.
• Lead the development of CI/CD and MLOps processes to automate deployments, monitoring, and model lifecycle management.
• Develop and maintain applications for aggregating, processing, and analyzing data from diverse sources, ensuring efficiency and scalability.
• Collaborate with Data Science teams on Machine Learning projects, including text/image analysis, feature engineering, and predictive model deployment.
• Design and manage complex data transformations using Databricks, dbt, and Apache Airflow, ensuring data integrity and consistency.
• Translate business requirements into scalable and efficient technical solutions while ensuring optimal performance and data quality.
• Ensure data security, compliance, and governance best practices are followed across all data pipelines.