A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
Repository Index
78,850 visible repositories in the index
169–192 of 221 · Most stars
Simple web scraping for R
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
A chat bot for Slack (https://slack.com).
This is a lightweight, headless browser purpose-built for AI agents (such as Codex or Claude Code) to perform browser automation tasks; it is designed to navigate through various web protection systems, login processes, and reverse-engineering scenarios
- anti-bot
- automation
- firefox
- headless-browser
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
- meta-search-engine
- rust
- search-engine
- fast
源码级 Chromium 指纹内核，检测站按普通 Chrome 评分。Source-level Chromium fingerprint kernel that passes bot detection tests. Not JS injection — C++ patches compiled into the binary.
- ai-agents
- anti-detect
- anti-detect-browser
- bot-detection
Crawly, a high-level web crawling & scraping framework for Elixir.
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
Clone Discord servers in real time with a powerful set of tools: export messages and DMs, scrape member lists, forward messages to other services, and much more. Controlled with slash commands or a web dashboard.
- discord
- discord-cloner-bot
- discord-exporter
- discord-message-sender
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
- agpl
- ai-agents
- anti-bot
- crawling
A high performance web crawler / scraper in Elixir.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Java API For Chrome and Firefox
:scissors: High performance, multi-threaded image scraper
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
- automation
- ecommerce
- filamentphp
- laravel
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands.
The unofficial HLTV Node.js API
High-performance HTML5 parser for Ruby based on Lexbor, with support for both CSS selectors and XPath.
Functional HTML scraping and rewriting with CSS in OCaml
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
The unix-way web crawler