Senior Python Data Scraping Engineer (Freelance)

Mindrift connects specialists with AI projects from major tech innovators, focusing on unlocking the potential of Generative AI by tapping into real-world expertise globally. This role is part of the Tendem project, which operates a hybrid AI + human system.

• Minimum 5 years of experience in data engineering, web scraping, automation, or software development.
• Strong technical foundation with practical experience in scripting, automation, and AI-assisted workflows.
• Proficient in Python web scraping with tools like BeautifulSoup and Selenium, including handling dynamic content (JavaScript, AJAX, infinite scroll) and APIs using proxies.
• Proven ability to extract data from complex and inconsistent web structures.
• Experience with data cleaning, normalization, and delivering structured datasets in formats such as CSV, JSON, and Google Sheets.
• Experienced in handling anti-bot mechanisms and dynamic site structures at scale.
• Familiarity with cloud infrastructure (AWS or equivalent) and containerization (Docker).
• Hands-on experience with large language model frameworks like LangChain and OpenRouter applied to automation tasks.
• Strong attention to detail and ability to work independently.
• English proficiency at Upper-intermediate (B2) level or above.

• Own end-to-end data extraction workflows across complex websites, ensuring completeness, accuracy, and reliable delivery of structured datasets.
• Utilize internal tools like Apify, OpenRouter alongside custom workflows to accelerate data collection, validation, and task execution.
• Adapt scraping approaches to reliably extract data from dynamic and JavaScript-rendered web content.
• Enforce data quality standards through validation checks and cross-source consistency controls.
• Scale scraping operations for large datasets using batching or parallelization, monitor failures, and maintain stability despite site changes.