🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- pocket
- wget
- browser-bookmarks
- pinboard
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
- javbus
- avmoo
- javlibrary
- spider
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
This is a python API which allows you to get the transcript/subtitles for a given YouTube video. It also works for automatically generated subtitles and it does not require an API key nor a headless browser, like other selenium based solutions do!
The Unofficial TikTok API Wrapper In Python
- tiktok-api
- python
- api
- trending
TranslatedZotero translators for Chinese web page scraping 🎉 This is Zotero translators for Chinese Sites(beta), not the official Zotero repo
- zotero
- translators
- javascript
- cnki
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单,功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
Export tweets, bookmarks, lists and much more from Twitter(X) web app. (推文/书签/收藏/列表导出工具)
- twitter
- archiver
- exporter
- twitter-api
Scrape metadata from any URL using Open Graph, JSON-LD, HTML meta tags, and smart fallbacks.
- metadata
- article
- cheerio
- extract
Generate and download e-books from online sources.
Stock market analyzer and predictor using Elasticsearch, Twitter, News headlines and Python natural language processing and sentiment analysis
Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Turn any website to API by several clicks (serverless and support SPA!)
Find broken links, missing images, etc within your HTML.
Google Play Scraper for collecting public details about applications, games, movies & TV shows, books, and more.
- google
- google-play
- google-play-api
- google-play-scraper
The URL engine behind Iframely. Self-hosted parsers that turn links into metadata, previews and embeds.
- embeds
- oembed
- twitter-cards
- open-graph
A self hostable app that tracks prices and sends you notifications when prices match your preferences
- open-source
- price-comparison
- price-monitoring
- price-tracker
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
check links in web documents or full websites
BrowserWing turns your browser actions into MCP commands Or Claude Skill, allowing AI agents to control browsers efficiently and reliably. Say goodbye to slow, token-heavy LLM interactions — let agents call commands directly for faster automation. Perfect for AI-driven tasks, browser automation, and boosting productivity.
Fast proxy scraper and checker written in Rust. Collects HTTP, SOCKS4 and SOCKS5 proxies from any number of sources, verifies each one really works, and writes JSON and plain text with response time, exit IP, ASN and geolocation. Single binary for Windows, Linux, macOS, Android, FreeBSD, NetBSD, illumos and Solaris, with an interactive TUI.
- proxyscraper
- proxyscrape
- proxy-checker
- proxy