Senior Software Engineer (Web Scraping, Data acquisition, Data collection)

At NIQ Digital shelf, scraping involves processing tens of billions of data points daily from thousands of web and mobile sources, transforming them into actionable insights for major retailers and brands. The team consists of 60+ scrapers, engineers, analysts, and data specialists globally, collaborating closely with Customer Success, Operations, and Data teams to extract high-quality data efficiently. NIQ is the world’s leading consumer intelligence company, resulting from a 2023 merger with GfK, operating in over 100 markets and covering more than 90% of the global population.

• 5+ years of experience with Python and large-scale scraping projects.
• Deep understanding of anti-bot technologies, including fingerprinting, WAFs, and JavaScript challenges.
• Experience with Scrapy, Requests, httpx, Selenium, Playwright, or custom headless browser drivers.
• Skilled in handling proxies (residential, datacenter, rotating), session management, cookie injection, and TLS tweaking.
• Strong debugging skills in browser, HTTP traffic, and device-level interactions.
• Independent thinker who enjoys solving novel problems.
• Familiarity with structured storage formats like Parquet and Avro, or real-time stream processing.
• Knowledge of databases such as MongoDB, InfluxDB, and Redis, including optimization techniques.

Nice to have:

• Experience with real-device simulation for mobile scraping.
• Experience building and maintaining distributed task queues like Celery, Kafka.
• Experience scraping high-security platforms or mobile-first environments.
• Exposure to Linux or cloud environments (AWS, GCP).
• Basic understanding of Kubernetes, Cloud Run, and Cloud Functions for deployment and orchestration.

• Design and build distributed scraping architectures scalable across thousands of domains.
• Reverse engineer APIs and mobile apps using tools such as Frida, mitmproxy, Charles Proxy, and burp.
• Detect anti-bot measures such as JavaScript checks or browser behavior detection to create human-like crawlers.
• Lead technical innovation in scraping by identifying edge cases and prototyping new approaches.
• Set up anomaly detection and monitoring to detect blockers or failures early.

NIQ may utilize artificial intelligence tools in the recruitment process to improve efficiency, with final hiring decisions made exclusively by humans. NIQ is committed to diversity, equity, and inclusion, making employment decisions without regard to protected characteristics under applicable law.