Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Repository Index
47,133 repositories indexed
1–11 of 11 · Most stars
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
PHP Magic Number Detector
🗑️ Easily remove the node_modules folder of non-active projects
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
Rust web scraper & crawler for AI agents: URLs, PDFs and Office files → clean Markdown/JSON. Firecrawl- and Jina Reader-compatible API, MCP server, Python/Node SDKs. Self-hosted single binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
TranslatedDeepSeek Blue Big Fat Fish Elimination Plan: a Chrome / Edge / Firefox browser extension that identifies and cleans up bot accounts (大肥鱼, Dafeiyu, "big fat fish") driven by the DeepSeek large model and AI agents in Bilibili comment sections. Offline rules + precise judgment by the DeepSeek API, AI comment detection, profile page identification, and one-click clearing by 鲸鱼娘 (Jingyuniang, "whale girl"). DeepSeek AI bot detector for Bilibili comments.
- ai-detection
- bilibili
- browser-extension
- chrome-extension
🚧 go-lambda-cleanup - A solution for removing previous versions of AWS Lambdas
Open Source & Self-hosted Spotify release tracker. Auto-syncs new releases to playlists and includes power-user utilities like a discography generator and ghost track cleaner.
- automation
- django
- django-rest-framework
- docker
Transform Git history: rewrite, clean, and optimize your repositories with ease.
- automation
- cli
- git
- python