Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 85 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Python scraper based on AI
PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Turn any website into a structured API. Extract, automate, search and monitor the web.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
A fast, helpful, and open-source document parser
OCR & Document Extraction using vision models
PyMuPDF is a high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents.
Plumb a PDF for detailed information about each char, rectangle, line, et cetera — and easily extract text and tables.
The fast, precise document-intelligence engine for every language: text, tables and metadata from 100+ formats, with OCR, transcription and code intelligence.
Automatic extraction of relevant features from time series:
The macOS & iOS file archiver
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
An NLP library for building bots, with entity extraction, sentiment analysis, automatic language identify, and so more
Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications
State of the Art Natural Language Processing
a library for audio and music analysis
Module for automatic summarization of text documents and HTML pages.
Malware Configuration And Payload Extraction
A library for audio and music analysis, feature extraction.