Software Engineer - Data Platforms (Databricks)

Acaisoft cooperates with a US client from the BioPharma sector delivering cloud software and services to improve the lives of patients by unlocking the potential of AI/ML.

• Strong hands-on experience with Python and complex SQL, and production Spark experience on Databricks including Structured Streaming, Auto Loader, Delta Lake, and Unity Catalog.
• Experience working with structured and unstructured data through ingestion, transformation, enrichment, indexing, and retention using engineering best practices such as infrastructure-as-code, CI/CD, and automated pipeline testing.
• Proven ability to ingest data from heterogeneous operational sources including relational (Oracle, PostgreSQL) and document stores (MongoDB), using Full load + CDC or equivalent replication patterns, handling schema drift, deletes, late or out-of-order data.
• Skills in data modelling across layered architecture with clear contracts between raw, curated, and serving layers supporting cross-entity queries and performant data access.

Nice to have:

• Experience building unstructured document pipelines including parsing and extraction from PDF, Office, and scanned formats, chunking, embedding generation, and maintaining searchable indexes at scale.

• Build, evolve, and operate data ingestion and processing capabilities for structured, semi-structured, and unstructured data, supporting transition from early prototypes to general release.
• Implement and maintain rich metadata and data quality practices enabling cross-record querying, traceability, and AI-ready data access.
• Collaborate with architects, AI engineers, workflow engineers, and domain experts to ensure data usability, performance, and trustworthiness.
• Support engineering quality through code reviews and mentoring of junior/mid-level engineers.

Due to the client's location, the team works till 6:00 PM CEST.