DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Repository Index
47,133 repositories indexed
25–48 of 53 · Most stars
Headless Chrome .NET API
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Find web directories without bruteforce
Async Python 3.6+ web scraping micro-framework based on asyncio
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
Crawly, a high-level web crawling & scraping framework for Elixir.
A scalable, mature and versatile web crawler based on Apache Storm
A high performance web crawler / scraper in Elixir.
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
Java API For Chrome and Firefox
✈️ 自托管的多渠道机票价格监控：航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线（五洲目的地）同一套采集-判定-推送链路；数据留在本机，配置全在网页，开源免费。
The unix-way web crawler
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Extract web archive data using Wayback Machine and Common Crawl
Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
Pre-rendering for JavaScript websites that delivers SSR-level SEO, enhanced link previews, and performance via effortless middleware integration — ideal for PWAs, SPAs, and modern JS-driven apps, websites, and webpages
Practical pinned gist which show your latest medium status 📌
SpiderMan,a base-on Broadway fast high-level web crawling & scraping framework for Elixir.