Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Repository Index
142,273 visible repositories in the index
313–336 of 627 · Most stars
收集各种免费的 Python 爬虫项目
- crawler
- python
- requests
- scrapy
A chat bot for Slack (https://slack.com).
Simple script for downloading Youtube comments without using the Youtube API
- youtube-comments
- youtube
- python
- data-science
批量抓取AV磁链或封面的苦劳力
modular service framework to move and transform network packets
源码级 Chromium 指纹内核,检测站按普通 Chrome 评分。Source-level Chromium fingerprint kernel that passes bot detection tests. Not JS injection — C++ patches compiled into the binary.
- ai-agents
- anti-detect
- anti-detect-browser
- bot-detection
HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.
- proxy
- proxies
- https-proxy
- proxy-server
A browser extension that captures web pages to local device or backend server for future retrieval, organization, annotation, and edit. This project inherits from legacy Firefox add-on ScrapBook X.
- capture
- archive
- maff
- bookmark
A tool to scrape LinkedIn without API restrictions for data reconnaissance
DOM Traversing and Scraping using GraphQL
101 real world web scraping exercises in Python 3 for data journalists
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
- meta-search-engine
- rust
- search-engine
- fast
A completely revamped and redesigned fork with OnlyFans and Fansly support, reimagined from scratch based on the original onlyfans-scraper.
- datascraping
- downloader
- fansite
- onlyfans
Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.
an unofficial screen-scraping "API" for Mint.com
A JavaScript library for generating random user agents with data that's updated daily.
- user-agent
- user-agent-spoofer
- random
- randomization
Provide scrapers for all major Israeli banks and credit card companies
World’s single largest Internet domains dataset
- yacy
- scrapy
- dataset
- search-engines
Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
- competitive-programming
- automation
- scraping
- atcoder
A self-hosted API that takes a URL and returns a file with browser screenshots.
📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.
- articles
- articles-data
- crawler
- datasets-preparation
python爬虫
- python
- python3
- urllib
- requests
First ever tool to view "Instagram private posts" anonymously