Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Repository Index
47,133 repositories indexed
1–24 of 29 · Most stars
Make Any Website into CLI & Use your logged-in browser by AI agent.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Capture website screenshots
The RSS feed for websites missing it
- php
- rss-bridge
- unlicense
- atom-feed
✨ The all-contributors bot website and documentation. Recognize all contributors, not just the ones who push code ✨
- contributors
- all-contributors
- opensource
- opensource-management
A Python module to bypass Cloudflare's anti-bot page.
- cloudflare
- anti-bot-page
- protected-page
- scrape
Fast website link checker in Go
💪 User-generated content for Git-powered websites
- static-site-generator
- jekyll
- hugo
- middleman
Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.
Live Helper Chat - live support for your website. Featuring web and mobile apps, Voice & Video & ScreenShare. Supports Telegram, Twilio (whatsapp), Facebook messenger including building a bot.
- live-support
- chat-application
- chat
- livehelp
YC F26. Turn any website into an API / MCP in minutes. Open source core integration minting harness. MIT.
WKZombie is a Swift framework for iOS/OSX to navigate within websites and collect data without the need of User Interface or API, also known as Headless browser. It can be used to run automated tests / snapshots and manipulate websites using Javascript.
Crawly, a high-level web crawling & scraping framework for Elixir.
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
- automation
- ecommerce
- filamentphp
- laravel
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.
A curated list of awesome nocode and lowcode ressources for building software without code.
Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.
- rvest
- rselenium
- httr
- scrape-websites
Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.