A multi-thread crawler framework with many builtin image crawlers provided.
- crawler
- python
- scrapy
- google-images
142,273 visible repositories in the index
385–408 of 627 · Most stars
A multi-thread crawler framework with many builtin image crawlers provided.
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
source for Open States scrapers
Collection of scripts corresponding to LucidProgramming YouTube tutorials
Rails backend for a scraper that outputs World Cup data as JSON
闲鱼爬虫,基于闲鱼网页端开发,使用 FastAPI 构建的闲鱼商品搜索接口,支持异步并发请求和自动化数据去重存储。现已支持登录。
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
Bulk download your favourite anime episodes from your favourite anime websites
Instagram Scraper. Scrape useful data/posts from instagram users, hashtag and locations pages. Comments and people who liked specific posts and soon more. No login or API keys are required
mmm lobster
A complete Private Instagram API for .NET (C#, VB.NET).
HTTP API for Scrapy spiders
An open database of international sanctions data, persons of interest and politically exposed persons
Non-official client to get some info about products sold on Amazon
MetaTube SDK & API Server in Golang
Tool for automated TikTok interactions
Free open-source Multilogin/Incogniton/Kameleo alternative for fingerprint spoofing (Canvas/WebGL/User-Agent). Perfect for Selenium/Playwright/Puppeteer automation, web scraping, social media bots, ads verification, and multi-accounting based on chromium. Avoid bans, bypass anti-bot systems, and prevent tracking with customizable stealth browsing
Generate Free Edu Mail(s) within minutes
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
Make a ZIM file from any Web site and surf offline!
Official Agent skills of Oxylabs products
invisible playwright mcp: Playwright MCP server undetected by anti-bots and captchas: AI agent browses the web on anti-detect stealth Firefox, Python, undetected browser automation, scraping, computer use.
Web Application Firewall (WAF) for PHP.
Crawl BookCorpus