The next web scraper. See through the <html> noise.
Repository Index
47,133 repositories indexed
25–48 of 63 · Most stars
A Python module to bypass Cloudflare's anti-bot page.
- cloudflare
- anti-bot-page
- protected-page
- scrape
Scrape Twitter for Tweets
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Crawly, a high-level web crawling & scraping framework for Elixir.
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
- agpl
- ai-agents
- anti-bot
- crawling
A high performance web crawler / scraper in Elixir.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Java API For Chrome and Firefox
:scissors: High performance, multi-threaded image scraper
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
- automation
- ecommerce
- filamentphp
- laravel
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
The Swiss Army knife to power JavaScript based client solutions ranging from Web apps/explorers over browser extensions to server-side clients like faucets/scrapers.
Semrush SEO AI-powered analysis tool MCP server & CLI for keyword research, domain analytics, backlink analysis
- backlink-analysis
- claude-desktop
- cursor-pro
- semrush
The unofficial HLTV Node.js API
A dataset of millions of news articles scraped from a curated list of data sources.
Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.
- rvest
- rselenium
- httr
- scrape-websites
TranslatedAn Agent Skill for scraping the complete comment section of a single Douyin video (top-level comments + nested replies): it copies the page's real requests inside a logged-in Chrome and leaves the signing to the page's SDK
- agent-skill
- claude-code
- comments-scraper
- douyin
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
DocSearch - Scraper
- algolia
- docsearch
- documentation
- scraper
Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.