Lead Engineer - Alternative Data & Web Extraction (Python / AI)

On behalf of our client - a premier, global US-headquartered financial technology and proprietary market analytics enterprise operating at massive scale. They operate engineering hubs delivering market-defining technological capabilities powered by big data, distributed systems, and applied machine learning. The Warsaw engineering group focuses on designing high-throughput systems that ingest, transform, and structure petabytes of alternative data from web, digital platforms, multimedia, and APIs for algorithmic decision engines and research platforms.

• 7–10+ years of commercial software engineering or data engineering experience with leadership or management of engineering teams.
• Bachelor's or Master's degree in Computer Science, Software Engineering, Mathematics, Physics, or related STEM discipline.
• Expert-level proficiency in Python, including libraries like Pandas and NumPy, and advanced web scraping/extraction toolkits.
• Strong knowledge of SQL, relational and non-relational database design, network protocols, web standards (HTML, DOM, REST/GraphQL APIs, browser automation), and anti-scraping mitigation.
• Practical familiarity with modern agentic workflows such as Claude Code, MCPs, subagents, and LLM-assisted data extraction and parsing.
• Solid experience with data streaming and pipeline orchestration using tools such as Apache Airflow and Kafka.
• Excellent verbal and written English (C1 or higher) with strong interpersonal and stakeholder management skills.

Nice to have:

• Experience in high-scale FinTech, high-throughput digital platforms, or systems serving data to Quant/Data Science teams.
• Hands-on experience with containerization and orchestration tools like Docker and Kubernetes, and modern cloud infrastructure (preferably AWS).
• Experience architecting data validation loops and managing human-in-the-loop validation teams.

• Lead, coach, and support a talented group of engineers, establish coding standards, and conduct code reviews.
• Architect resilient, scalable, and low-latency extraction frameworks for massive unstructured datasets from web, APIs, and multimedia.
• Design and deploy agent-driven pipelines leveraging LLMs, Model Context Protocols (MCPs), subagents, and automated workflows.
• Build and maintain fault-tolerant streaming and orchestration architectures using queueing systems like Kafka, workflow engines such as Airflow, and relational/NoSQL datastores.
• Partner with data researchers, quantitative analysts, and domain experts to deliver production-grade data products.
• Collaborate with distributed validation teams to build human-in-the-loop workflows for dataset verification.
• Track emerging trends in anti-bot mitigation, browser automation, and LLM-powered extraction technologies.

I consent to the processing of my personal data by Yard Corporate Sp. z o.o. and its affiliated entities operating in recruitment, HR technology, talent platforms or professional matching services for recruitment purposes as described. This consent is valid for 36 months and may be withdrawn at any time.