Lead Data Engineer (Databricks)

The position is for a Lead Data Engineer collaborating with a US-based BioPharma organization. The client delivers specialized cloud software and services for the BioPharma sector, aiming to leverage AI/ML to improve patient outcomes.

• Experience as a Lead / Principal Data Architect in multinational environments.
• Deep knowledge of data modelling, integration, warehousing, and modern data platforms.
• Strong hands-on experience with Python and complex SQL, and production Spark experience on Databricks (including Structured Streaming, Auto Loader, Delta Lake, Unity Catalog).
• Practice working with structured and unstructured data across the full lifecycle: ingestion, transformation, enrichment, indexing, and retention, following engineering best practices (infrastructure-as-code, CI/CD, automated pipeline testing).
• Proven experience ingesting data from heterogeneous sources including relational databases (Oracle, PostgreSQL) and document stores (MongoDB), using full load plus CDC or equivalent replication patterns.
• Skills in data modelling across layered architectures with clear contracts between raw, curated, and serving layers supporting cross-entity queries and performant access patterns.

• Build, evolve, and operate data ingestion and processing capabilities for structured, semi-structured, and unstructured data.
• Implement and maintain rich metadata and data quality practices enabling cross-record querying and AI-ready data access.
• Collaborate with architects, AI engineers, workflow engineers, and domain experts to ensure data usability, performance, and trustworthiness.
• Support engineering quality through code reviews and mentoring junior/mid-level engineers.
• Lead architectural decisions and engage with senior business and technology stakeholders.

Nice to have:

• Experience building unstructured document pipelines including parsing and extraction from PDF, Office, scanned formats, chunking, embedding generation, and maintaining searchable indexes at scale.

Due to the client's location, the team works until 6:00 PM CEST.