Data Engineer

Scalo delivers software projects and supports partners in business development. They utilize wide IT services including software creation, embedded systems, cloud solutions, data management, technology consulting, and dedicated project teams.

• Experience in data engineering, specifically building pipelines and data processing services (3-5 years)
• Good knowledge of Python or Rust
• Practical knowledge of vector databases, especially Qdrant
• Familiarity with embeddings, semantic search, indexing, and metadata filtering
• Experience designing or developing REST API integrations
• Knowledge of event-driven architecture, e.g., Kafka or Pub/Sub
• Good knowledge of Linux system
• Knowledge of relational databases, e.g., PostgreSQL
• Ability to design and develop data processing workflows
• Independence in analyzing technical problems and proposing solutions

Nice to have:

• Experience with Docker, Kubernetes, and CI/CD
• Knowledge of GCP, including BigQuery, Dataflow, Dataproc or Pub/Sub
• Familiarity with Apache Airflow, Apache Spark, or Apache Beam
• Experience processing unstructured data including OCR, audio transcription, and document conversion
• Knowledge of graph databases, e.g., Neo4j
• Experience with Knowledge Graph, GraphRAG, or classical RAG solutions
• Familiarity with FastAPI

• Build and develop data pipelines processing unstructured data for an AI project
• Design and implement ETL flows for an Unstructured Data Repository and AI agent solutions
• Create data processing workflows for documents, images, and audio files
• Implement content extraction mechanisms, OCR, ASR (audio transcription), and document conversion
• Develop solutions using embeddings, semantic search, indexing, and metadata filtering
• Integrate pipelines with vector databases, especially Qdrant
• Design and develop data processing services with Python or Rust
• Create and integrate services providing functionality via REST API
• Develop event-driven data processing mechanisms using Kafka or Pub/Sub
• Collaborate in designing data categorization services, ontologies, and AI information processing solutions
• Analyze and optimize performance and reliability of data processing workflows
• Collaborate with Data Engineering and AI teams

The work model is hybrid with one day per week onsite in the Warsaw office.