Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Repository Index
142,273 visible repositories in the index
1–24 of 33 · Most stars
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- wget
- browser-bookmarks
- pinboard
⬛️ GUI/CLI tool and library for saving complete web pages as a single HTML file
- save-the-internet
- procrastination
- its-mine
- no-more-404
Turn any webpage into structured data using LLMs
Distributed crawler powered by Headless Chrome
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
Mechanize is a ruby library that makes automated web interaction easy.
- scraping
- web
- ruby
job-ops: DevOps principles applied to job hunting. A self-hosted pipeline to track, analyze, and assist your application process
- automation
- self-hosted
- job-search
- jobhunt
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- endpoints
- endpoint-discovery
- bugbounty
- crawler
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
IPTV 国内+国外 电视台直播源m3u文件, 收集&汇总&本地源脚本
- iptv
- m3u
- python
- m3u8-playlist
Auto-bypass for blocked websites in Claude Code — Phase 0→3 adaptive scheduler, no API keys
Download images from Pixiv and more!
A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript
- playwright
- playwright-python
- scraping
- screenshot-utility
AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project
- ail-framework
- information-leak
- information-security
- analysis
一个功能强大的自托管弹幕(Danmaku)聚合与管理服务,兼容 弹弹play API 规范。
DOM Traversing and Scraping using GraphQL
A self-hosted API that takes a URL and returns a file with browser screenshots.
🛡️ Автоматический сборщик MTProto‑прокси с уникальной фильтрацией «Умный комбайн» для автоматического сбора, анализа и фильтрации MTProto‑прокси. Отделяет **"золотые" прокси**, маскирующиеся под Yandex, VK, Gosuslugi и другие популярные RU‑сервисы (Fake‑TLS), от обычных.
FanFicFare is a tool for making eBooks from stories on fanfiction and other web sites.
- fanfiction
- epub
- ebook
- downloader
TweetScraper is a simple crawler/spider for Twitter Search without using API
- twitter-search
- scrapy
- tweets
AIL framework - Analysis Information Leak framework
- ail-framework
- information-extraction
- information-security
- data-mining
A next-generation HTTP stealth proxy which perfectly cloaks requests as the Chrome browser across all layers of the stack.
"loklak wok" is a harvesting peer for the loklak_server