Senior Webscraping & Data Engineer

• 6+ years of experience in software or data engineering.
• Strong experience with web scraping, crawling, and automated data collection.
• Excellent Python skills, including pandas.
• Strong SQL and database knowledge.
• Good understanding of HTML, JavaScript, APIs, and web technologies.
• Knowledge of networking concepts related to web data collection.
• Experience building data pipelines.
• Understanding of algorithms, data structures, and data quality.
• Experience with large-scale or distributed data collection.
• Strong troubleshooting and problem-solving skills.
• Interest in AI/LLMs and agentic technologies.
• Strong communication and collaboration skills.

Nice to have:

• Experience with alternative data in finance or research.
• Background in data-intensive organizations.
• Experience building tools for researchers, analysts, quants, or data scientists.
• Knowledge of Airflow, Kafka, Docker, or Kubernetes.
• Experience with AWS or other cloud platforms.
• Familiarity with LLM tools, MCPs, agents, or autonomous workflows.
• Experience with audio/video data.
• Knowledge of statistics and data analysis.

• Design and develop scalable web scraping and data collection systems.
• Build scrapers and crawlers for websites with different structures and technologies.
• Collect data from websites, APIs, and dynamic online sources.
• Clean, transform, and standardize raw data.
• Improve accuracy, speed, coverage, and reliability of scraping solutions.
• Troubleshoot website changes, data quality, and performance issues.
• Build automated pipelines for processing collected data.
• Explore AI/ML and agentic technologies for extraction and validation.
• Support infrastructure for large-scale data collection.
• Implement monitoring and data quality controls.
• Work with researchers and analysts to understand data needs.
• Explore new technologies and hard-to-source data.