Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 226 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Turn any PDF or image document into structured data for your AI. A powerful, lightweight OCR toolkit that bridges the gap between images/PDFs and LLMs. Supports 100+ languages.
- ocr
- chineseocr
- pdf2markdown
- pp-ocr
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Web Extension for saving a faithful copy of a complete web page in a single HTML file
- browser
- archive
- auto-save
- chrome
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
Replace 'hub' with 'ingest' in any GitHub URL to get a prompt-friendly extract of a codebase
- ai
- code
- ingestion
- developer-tool
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
Extract and decrypt browser data, supporting multiple data types, runnable on various operating systems (macOS, Windows, Linux).
- chrome
- macos
- windows
- pentest-tool
YouTube Music client for Android
- android
- innertube
- material-design
- material-ui
HTTP API design guide extracted from work on the Heroku Platform API
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
A fast, helpful, and open-source document parser
- document-ocr
- document-processing
- ocr
- ocr-recognition
OCR & Document Extraction using vision models
PyMuPDF is a high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents.
- mupdf
- xps
- pdf-documents
- epub
Plumb a PDF for detailed information about each char, rectangle, line, et cetera — and easily extract text and tables.
- pdf-parsing
- table-extraction
Zero-runtime Stylesheets-in-TypeScript
Extract & Inline Critical-path CSS in HTML pages
- css
- critical-css
- critical-path-css
- critical-path-styles