Sign in

Repository Index

100,822 visible repositories in the index

Tags: Web scraper
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 36 · Most stars

TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database

  • javbus
  • avmoo
  • javlibrary
  • spider

TranslatedGet the real streaming URLs (live stream sources) and danmaku (live comments) of 58 live streaming platforms, including Douyu, Huya, Bilibili, Douyin, and Kuaishou. The live stream sources can be played in players such as PotPlayer and flv.js.

  • douyu
  • huya
  • bilibili
  • python

TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:

  • python3
  • crawler
  • baidu-tieba
  • taobao-spider

Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

  • scanner
  • crawler
  • information-gathering
  • admin-scanner

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

  • aitools
  • crawl
  • scrape
  • serp

🤖 Top-rated tools to scrape all major public sections from Facebook, Instagram, and Twitter (X) including posts (likes/comments), photos/videos, contact information, followers, following and much more.

  • facebook-page-insights
  • instagram-followers-scraper
  • facebook-ads-metrics
  • facebook-contact-info

WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.

Point it at any web page and it finds the video, transcodes it, and streams it to your TV in real time. It even burns in subtitles.

Retrieve author and publication information from Google Scholar in a friendly, Pythonic way without having to worry about CAPTCHAs!

  • scholar
  • googlescholar
  • scholarly-articles
  • scholarly-communications