Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 85 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
A fast, helpful, and open-source document parser
- document-ocr
- document-processing
- ocr
- ocr-recognition
OCR & Document Extraction using vision models
PyMuPDF is a high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents.
- mupdf
- xps
- pdf-documents
- epub
Plumb a PDF for detailed information about each char, rectangle, line, et cetera — and easily extract text and tables.
- pdf-parsing
- table-extraction
The fast, precise document-intelligence engine for every language: text, tables and metadata from 100+ formats, with OCR, transcription and code intelligence.
- text-extraction
- document-intelligence
- metadata-extraction
- pdf-extraction
Automatic extraction of relevant features from time series:
- data-science
- feature-extraction
- time-series
The macOS & iOS file archiver
- keka
- archiver
- macos
- osx
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
An NLP library for building bots, with entity extraction, sentiment analysis, automatic language identify, and so more
Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications
- audio
- machine-learning
- signal-processing
- audio-data
State of the Art Natural Language Processing
a library for audio and music analysis
Module for automatic summarization of text documents and HTML pages.
- python
- lsa
- textteaser
- html-page
Malware Configuration And Payload Extraction
- configs
- debugging-tools
- malware
- malware-analysis
A library for audio and music analysis, feature extraction.
- audio
- audio-analysis
- audio-features
- python