Senior Web Scraper (Scrapy)

At NIQ Digital shelf, scraping isn’t just about collecting data, it’s about outsmarting the obstacles that stand in the way of visibility. Every day, we process tens of billions of data points from thousands of sources, both web and mobile, and transform them into actionable insights for the world’s top retailers and brands. You'll join a team of 60+ scrapers, engineers, analysts, and data lovers from across the globe. This role is part of our scraping and R&D team, collaborating closely with Customer Success, Operations, and Data teams to extract high-quality, actionable data — fast and at scale.

• 5+ years of experience with Python and large-scale scraping projects.
• Deep understanding of anti-bot technologies, including fingerprinting, WAFs, and JavaScript challenges.
• Experience with Scrapy, Requests, httpx, Selenium, Playwright, or custom headless browser drivers.
• Skilled in handling proxies (residential, datacenter, rotating), session management, cookie injection, and TLS tweaking.
• Strong debugging skills across browser, HTTP traffic, and device-level interactions.
• Independent thinker who enjoys solving novel problems.
• Familiarity with structured storage formats like Parquet and Avro, or real-time stream processing.
• Good knowledge of databases such as MongoDB, InfluxDB, and Redis, including optimization.

Nice to have:

• Experience with real-device simulation for mobile scraping.
• Comfortable building and maintaining distributed task queues like Celery and Kafka.
• Experience scraping high-security platforms or mobile-first environments.
• Exposure to Linux or cloud environments (AWS, GCP).
• Basic understanding of Kubernetes, Cloud Run, and Cloud Functions for deploying and orchestrating scraping workloads.

• Design and build distributed scraping architectures that scale across thousands of domains.
• Reverse engineer APIs and mobile apps using tools like Frida, mitmproxy, Charles Proxy, burp.
• Detect anti-bot measures such as JavaScript checks and browser behavior detection to build human-like crawlers.
• Lead technical innovation in scraping by identifying edge cases, prototyping new approaches, and sharing knowledge.
• Set up anomaly detection and monitoring to detect blockers or failures early.

NIQ may utilize artificial intelligence (AI) tools during recruitment stages for tasks such as résumé screening, candidate assessments, and interview scheduling. Final hiring decisions are made exclusively by humans. NIQ is committed to fairness, transparency, human oversight, and inclusion in its AI usage. NIQ is an equal opportunity employer, making employment decisions without regard to protected characteristics.