Senior Python Data Scraping Engineer (Freelance)

Mindrift connects specialists with AI projects and focuses on unlocking the potential of Generative AI by integrating real-world expertise. This role is for their Tendem project, which blends AI and human workflows to drive specialized data scraping.

• Minimum 5+ years of experience in data engineering, web scraping, automation, or software development.
• Bachelor’s or Master’s degree in Engineering, Applied Mathematics, Computer Science, or related fields is a plus.
• Strong technical foundation in scripting, automation, and AI-assisted workflows.
• Experience with Python web scraping libraries (BeautifulSoup, Selenium or similar), handling dynamic content (JS, AJAX, infinite scroll), and APIs via proxies.
• Skills in extracting data from complex structures including hierarchies and inconsistent HTML.
• Background in data cleaning, normalization, and validation delivering structured datasets (CSV, JSON, Google Sheets).
• Experience addressing anti-bot measures and dynamic site structures at scale.
• Knowledge of cloud infrastructure (AWS or equivalent) and containerization (Docker).
• Hands-on experience with LLM frameworks such as LangChain and OpenRouter for automation.
• Strong attention to detail and ability to work independently.
• Upper-intermediate or higher English proficiency (B2 or above).

Nice to have:

• A GitHub profile link.

• Own end-to-end data extraction workflows across complex websites, ensuring complete coverage and accuracy.
• Use internal tools (Apify, OpenRouter) and custom workflows to accelerate data collection and validation.
• Adapt scraping methods to dynamic and interactive web content, including JavaScript-rendered sites.
• Implement data quality standards with validation checks, cross-source consistency controls, and format adherence.
• Scale scraping operations using efficient batching or parallelization and monitor for failures while maintaining stability.

This is a freelance, part-time remote opportunity. English proficiency at B2 or above is required. The workload varies and is estimated, not guaranteed.