A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Repository Index
78,850 visible repositories in the index
265–288 of 328 · Most stars
Java API For Chrome and Firefox
:scissors: High performance, multi-threaded image scraper
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
- automation
- ecommerce
- filamentphp
- laravel
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
The Swiss Army knife to power JavaScript based client solutions ranging from Web apps/explorers over browser extensions to server-side clients like faucets/scrapers.
Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands.
TranslatedAI coding perks navigation: a roundup of relay stations and public-service stations for Claude Code, Codex, and Cursor, organizing free quotas, sign-up bonuses, check-in activities, and evaluation records, and providing a visual dashboard and a Python data collection tool.
- ai-coding
- api-relay
- claude-code
- codex
TranslatedLet any AI agent control your own Chrome, with all your logged-in sessions. MCP + Chrome extension, works with Claude Code / Codex / Cursor / Gemini CLI.
- ai-agents
- browser-automation
- chrome-extension
- claude-code
Hand your SaaS growth chores to an agent: new-term radar, keyword-to-site discovery, competitor backlinks, directory submissions, plus a social analysis pack with a Douyin comments skill. Works with Grok Bot, Muse, Dot, Hermes, OpenClaw, Claude Code and Codex.
- agent-skill
- claude-code
- comments-scraper
- douyin
The unofficial HLTV Node.js API
EveryInfra MCP server & agent plugin: scraper API for TikTok, Instagram, Amazon, YouTube, Xiaohongshu & Douyin. 573 data APIs across 106 platforms, from $0.56/1K calls. For Claude Code, Cursor, Codex.
- ai-agents
- claude-code
- cursor
- mcp
Browser automation engine benchmark - Test bypass rates, performance & stealth against Cloudflare, DataDome, reCAPTCHA, Kasada, Imperva, Akamai, PerimeterX and other bot detection systems. Find the best browser for scraping.
- anti-detect
- antidetect-browser
- automation
- browser
High-performance HTML5 parser for Ruby based on Lexbor, with support for both CSS selectors and XPath.
A dataset of millions of news articles scraped from a curated list of data sources.
Functional HTML scraping and rewriting with CSS in OCaml
Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.
- rvest
- rselenium
- httr
- scrape-websites
Client-side media downloader App and CLI built with Capacitor and Node.js.
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
The unix-way web crawler
Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
DocSearch - Scraper
- algolia
- docsearch
- documentation
- scraper
TranslatedAI subscription and card-shop price comparison platform: say what you need and let AI help you choose. Aggregates top-up service or account offers for ChatGPT / Codex, Claude, Gemini and more, and compares price trends, top-up channels and warranties. AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.
- ai
- price-comparison
- self-hosted
- web-scraping
Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.