Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.
Repository Index
109,091 visible repositories in the index
1–24 of 759 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Turn any PDF or image document into structured data for your AI. A powerful, lightweight OCR toolkit that bridges the gap between images/PDFs and LLMs. Supports 100+ languages.
- ocr
- chineseocr
- pdf2markdown
- pp-ocr
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
A modern JavaScript utility library delivering modularity, performance, & extras.
- lodash
- utilities
- javascript
- modules
Convert PDF to markdown + JSON quickly with high accuracy
- deep-learning
- document-conversion
- document-parsing
- docx
A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.
Unlock your displays on your Mac. Flexible HiDPI scaling, XDR/HDR extra brightness, virtual screens, DDC control, image adjustments, PIP/streaming, EDID override. More info - betterdisplay.pro/guide
- hdmi
- mac
- resolution
- hidpi
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Web Extension for saving a faithful copy of a complete web page in a single HTML file
- browser
- archive
- auto-save
- chrome
Give Claude the ability to watch any video. /watch downloads, extracts frames, transcribes, hands it all to Claude.
Fast Rust library for PDF inspection, classification, and text extraction. Intelligently detects scanned vs text-based PDFs to enable smart routing decisions.
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
TranslatedThe Ultimate Guide to Making Money with AI Side Hustles: Learn how to leverage AI for some cool side gigs and rake in some extra cash. Check out the English version for more insights.
GIT utilities -- repo summary, repl, changelog population, author commit percentages and more
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
Replace 'hub' with 'ingest' in any GitHub URL to get a prompt-friendly extract of a codebase
- ai
- code
- ingestion
- developer-tool
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
Simple, powerful and flexible site generation framework with everything you love from Next.js.
- nextra
- nextjs
- static-site-generator
- mdx
HTTP API design guide extracted from work on the Heroku Platform API