🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- wget
- browser-bookmarks
- pinboard
100,822 visible repositories in the index
1–24 of 55 · Most stars
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
TranslatedDouyin / TikTok platform tool for downloading works and collecting data
⬛️ GUI/CLI tool and library for saving complete web pages as a single HTML file
Download pictures (or videos) along with their captions and other metadata from Instagram.
TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool
TranslatedAn online tool for batch downloading WeChat Official Account articles. It supports exporting read count and comment data, requires no environment setup, and can be used through an online website. It supports private deployment with docker and deployment on Cloudflare. It supports downloading various file formats, and the HTML format can restore article layout and styles 100%.
TranslatedA batch download tool for WeChat Official Account articles, supporting download of comments and collections, supporting saving html/mhtml/md/pdf/docx/csv files, saving images, videos, and audio files within articles, and supporting MCP/Skill invocation
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!
TranslatedPython API for JMComic | Provides a Python API for accessing 禁漫天堂 (JMComic), supporting both the web and mobile clients | 禁漫天堂 (JMComic) GitHub Actions downloader🚀
The Unofficial TikTok API Wrapper In Python
💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js
Download an entire website from the Wayback Machine.
TikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.
Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.
Datascraper that is currently adult focused (OnlyFans, etc)- Updated regularly
Downloads albums from the web in bulk for archive purposes
job-ops: DevOps principles applied to job hunting. A self-hosted pipeline to track, analyze, and assist your application process
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
Abstraction layer over YouTube's internal API
freehire — the open-source search engine for job seekers
Every web site provides APIs.