Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.
Repository Index
99,825 visible repositories in the index
1–24 of 46 · Most stars
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
API Support for your favorite torrent trackers
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
- apify
- automation
- beautifulsoup
- crawler
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
- golang
- google-maps-scraping
- web-scraper
- web-scraping
Turn any webpage into structured data using LLMs
Official Bright Data CLI - scrape, search, and extract structured web data directly from your terminal.
Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha
An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.
- ai-agent
- automation
- data-extraction
- event-driven-architecture
job-ops: DevOps principles applied to job hunting. A self-hosted pipeline to track, analyze, and assist your application process
- automation
- self-hosted
- job-search
- jobhunt
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
- aitools
- crawl
- scrape
- serp
Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
- amazon-scraper
- amazon-scraping
- e-commerce-api
- web-scraping
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
- amazon
- amazon-scraper
- python
- web-scraper
🤖 Top-rated tools to scrape all major public sections from Facebook, Instagram, and Twitter (X) including posts (likes/comments), photos/videos, contact information, followers, following and much more.
- facebook-page-insights
- instagram-followers-scraper
- facebook-ads-metrics
- facebook-contact-info
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
- search-engine
- python
- scraping
- crawler
Export tweets, bookmarks, lists and much more from Twitter(X) web app. (推文/书签/收藏/列表导出工具)
- archiver
- exporter
- twitter-api
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
- scraper
- sina
- corpus
Cloud browser infrastructure and web automation platform for your AI and coding agents
RSS-proxy allows you to do create an RSS or ATOM feed of almost any website, just by analyzing just the static HTML structure.
- atom-feed
- rss-generator
- rss-feed
- json-feed