Senior Software Engineer (Web Scraping, Data acquisition, Data collection)

132k - 233.4k PLN132 000 - 233 400 PLN/ rok./ rok.

SeniorHybryda·WarszawaFull-time

#475121·Dodano dziś·0

Źródło: NielsenIQ

Aplikuj teraz

Tech Stack / Keywords

PythonScrapyPlaywright SeleniumFridamitmproxyMongoDB Redis Kafka Kubernetes

Podsumowanie oferty

Build distributed web-scraping systems for NIQ Digital Shelf, collecting data across thousands of domains. The role focuses on designing scraping architectures, reverse engineering APIs and mobile apps, and detecting anti-bot measures, using Python and scraping tools such as Scrapy, Selenium, and Playwright.

Wymagania

5+ years of experience with Python and large-scale scraping projects.
Deep understanding of anti-bot technologies and how to bypass them, including fingerprinting, WAFs, and JavaScript challenges.
Experience with Scrapy, Requests, httpx, Selenium, Playwright, or custom headless browser drivers.
Skilled at handling proxies (residential, datacenter, rotating), session management, cookie injection, and TLS tweaking.
Strong debugging skills across browser, HTTP traffic, and device-level interactions.
Think independently and enjoy solving problems no one has solved before.
Familiarity with structured storage formats such as Parquet and Avro, or real-time stream processing.
Good knowledge of databases such as MongoDB, InfluxDB, and Redis, including how they work and how to optimize them.

Bonus Skills:

Experience with real-device simulation for mobile scraping.
Comfortable building and maintaining distributed task queues such as Celery and Kafka.
Experience scraping high-security platforms or mobile-first environments.
Exposure to Linux or cloud environments such as AWS and GCP.
Basic understanding of Kubernetes, Cloud Run, and Cloud Functions to deploy and orchestrate scraping workloads.

Obowiązki

Design and build distributed scraping architectures that can scale across thousands of domains.
Reverse engineer APIs and mobile apps using tools like Frida, mitmproxy, Charles Proxy, and Burp.
Detect common anti-bot measures, such as JavaScript checks or browser behavior detection, and build crawlers that adjust to appear more human-like.
Lead technical innovation in scraping by identifying edge cases, prototyping new approaches, and sharing knowledge with the team.
Set up anomaly detection and monitoring to detect blockers or failures early.

Benefity

Flexible working hours.
Volunteer time off.
LinkedIn Learning.
Employee Assistance Program (EAP).

Elastyczne godziny

NielsenIQ

15 aktywnych ofert

Zobacz wszystkie oferty

Aplikuj teraz