85,081 visible repositories in the index
1–24 of 165 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
🧡 Everything is RSSible
🕵️♂️ Collect a dossier on a person by username from 6K websites
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Crawl a site to generate knowledge files to create your own custom GPT from a URL
TranslatedA multi-task real-time/scheduled monitoring and intelligent analysis system for Xianyu (闲鱼), built with Playwright and AI, and equipped with a full-featured back-end management UI. It helps users find the products they like among the vast number of items on Xianyu.
Turn any website into a structured API. Extract, automate, search and monitor the web.
API Support for your favorite torrent trackers
⬛️ GUI/CLI tool and library for saving complete web pages as a single HTML file
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
A little like that j-thing, only in Go.
Download pictures (or videos) along with their captions and other metadata from Instagram.
Pythonic HTML Parsing for Humans™
Incredibly fast crawler designed for OSINT.
Python APIs for web automation, e2e testing, and bypassing bot-detection with ease.
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
A standalone version of the readability lib
Get the main content of any page as Markdown.
The RSS feed for websites missing it
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.