Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
Repository Index
78,850 visible repositories in the index
193–216 of 221 · Most stars
DocSearch - Scraper
- algolia
- docsearch
- documentation
- scraper
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library
- golang
- unmarshall
- goquery
- selectors
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
- golang
- web-crawler
- crawling
- scraping
Playwright PHP library for browser automation: navigation, E2E tests, assertions, screenshots, and so much more!
- browser
- browser-automation
- php
- php-browser
Self-hosted, invisible CAPTCHA that detects AI agents driving real browsers. GDPR, WCAG 2.2 AA, and EAA: no cookies, no third-party data sharing, and no visual or audio puzzle. With a false-positive budget enforced in CI. Proof of work, drop-in for Turnstile and reCAPTCHA.
- anti-bot
- behavioral-analysis
- bot-detection
- captcha
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
🔥 A daily streak system & portfolio for your projects.
- summer-of-making
- nextjs
- hackclub
- hacktoberfest
Search and read the web for AI agents at 99% fewer tokens. Free CLI + MCP.
MindAct: Community-Driven AI Agent for Claude, Obsidian, and RAG Knowledge Graphs
- activator
- analytics
- artificial-inteligence
- bigquery
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
- go
- golang
- goquery
- colly
An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.
Scrapy project for data capture of vgchartz
The AI-native browser automation library. Snapshot + ref targeting — born from OpenClaw, built for agents, by agents.
Search the web and primary sources, read full pages, and check every citation. MCP server for Claude, Cursor, Codex and more. Runs on your machine.
Webpage summary extractor using Facebook Open Graph and arc90's readability
- readability
- scraper
- opengraph
Website to Scrapping all writeup from http://ctftime.org/ and you can organize which to read first!
Browser extension to simulate browsing behaviour in search engines.
- automation
- audit
- browser-extension
- firefox
Community-driven Discord server cloner with a full installer and ready-to-run desktop app for effort
- automation
- developer-tools
- discord-channel-cloner
- discord-clone
Go package for scraping website recipes
- go
- golang
- recipe
- recipes
Text Generation using Markov Chains fed by 4chan APIs
- scrapper
- data-mining
- markov-chain
SpiderMan,a base-on Broadway fast high-level web crawling & scraping framework for Elixir.
Seamlessly fetch paginated data from any source. Simple and high performance API scraping included!
- api-scraper
- indexer
- pagination
- paginator