Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.
Repository Index
111,085 visible repositories in the index
1–24 of 30 · Most stars
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
A standalone version of the readability lib
SEX IS ZERO (0), so, who wanna be the ONE (1), aha?
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
- golang
- google-maps-scraping
- web-scraper
- web-scraping
One-stop Proxies Crawling and Aggregation Platform
- proxypool
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.
Extract any website’s design system into tokens in seconds: logo, colors, typography, borders & more. One command.
Google Maps Scraper & Lead Generation Tool. Extract 50+ data points including business emails, phone numbers, and social profiles. Includes enrichment features, API access, and a generous free plan with 200 searches/month.
- google-maps-scraping
- google-maps
- google-maps-api
- google-maps-review
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
- scanner
- crawler
- information-gathering
- admin-scanner
Watch (parts of) webpages and get notified when something changes via e-mail, on your phone or via other means. Highly configurable.
- python
- webpage
- monitor
- automation
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
- b2b-leads
- claude-code
- google-maps
- google-maps-scraper
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
- golang
- go
- cloning
- crawler
Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
QQ空间导出助手,用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享、最近访客为文件,便于迁移与保存
- chrome
- qqzone
- export
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
微博爬虫及配套工具箱,微博用户、话题、评论采集一网打尽。图片下载、情感分析,地理位置、关系网络、spammer 机器人识别等功能应有尽有。Docs:https://buyixiao.github.io/blog/weibo-super-spider.html 配套可视化网站:https://buyixiao.github.io/blog/one-stop-weibo-visualization.html
- weibo-spider
- weibocrawler
- weibo-comment-crawl
- emotion-analysis
This code is used to perform web scraping and data extraction from Google Maps. It is particularly designed for obtaining information about businesses, including their name, address, website, phone number, reviews, and more.
- chromium
- data
- data-scraping
Anemone web-spider framework
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.
- data-science
- politics
- open-data
- civic-tech