Senior Python Data Scraping Engineer (Freelance)

Mindrift connects specialists with AI projects and aims to unlock the potential of Generative AI by leveraging domain expertise globally. This role is part of the Tendem project, focused on specialized data scraping workflows within a hybrid AI + human system.

• Minimum 5 years of experience in data engineering, web scraping, automation, or software development.
• Strong expertise in Python web scraping with tools such as BeautifulSoup, Selenium, including experience with dynamic content (JavaScript, AJAX, infinite scroll) and API usage via proxies.
• Proven ability to extract data from complex and inconsistent web structures.
• Experience in data cleaning, normalization, validation, and delivering structured data formats like CSV, JSON, and Google Sheets.
• Handling anti-bot mechanisms and dynamic site structures at scale.
• Familiarity with cloud infrastructure such as AWS and containerization using Docker.
• Hands-on experience with LLM frameworks like LangChain and OpenRouter for automation.
• Upper-intermediate (B2) English proficiency or above.
• Self-directed work ethic with strong attention to detail and problem-solving skills.

Nice to have:

• Bachelor’s or Master’s degree in Engineering, Applied Mathematics, Computer Science, or related fields.
• Experience working confidently with large language models (LLMs).
• A GitHub portfolio link.

• Own end-to-end data extraction workflows ensuring complete coverage, accuracy, and structured datasets.
• Use internal tools (Apify, OpenRouter) and custom workflows for data collection, validation, and task execution.
• Adapt scraping methods to handle dynamic and JavaScript-rendered content.
• Enforce data quality through validation checks, consistency controls, and adherence to formatting.
• Scale scraping operations via batching or parallelization and monitor workflow stability.