Sign in

Repository Index

132,241 visible repositories in the index

Tags: All Tags
Tags

A repository shows when it has any selected Tag.

Filters

Filters

Clear all filters

1–24 of 286 · Most stars

TranslatedA visual no-code/code-free web crawler/spider. 易采集 (EasySpider, "easy collection"): a visual browser automation testing/data collection/web crawler software that lets you design and execute crawler tasks graphically without code. Alias: ServiceWrapper, an intelligent service wrapping system for Web applications.

TranslatedA Python web scraping tutorial series, learning Python web scraping from 0 to 1, including browser packet capture and mobile APP packet capture with tools such as fiddler and mitmproxy, the use of the various modules involved in web scraping, such as requests, beautifulSoup, selenium, appium, scrapy, etc., as well as IP proxies, CAPTCHA recognition, using Mysql and MongoDB databases with Python, multi-threaded and multi-process scraping, reverse engineering and cracking of CSS scraping encryption, JS scraping reverse engineering, distributed scraping, practical scraping project examples, etc.

  • python3
  • python-script
  • python-spider

Translated:rainbow:Python3 web crawler practice: Taobao, JD.com, NetEase Cloud Music, Bilibili, 12306, Douyin, Biquge, comic and novel downloads, music and movie downloads, etc.

  • python-spider
  • python3
  • python
  • webspider

TranslatedSome interesting examples of python crawlers that are friendly to beginners. Mainly crawls websites such as 淘宝 (Taobao, "searching for treasure"), 天猫 (Tmall, "sky cat"), 微信 (WeChat, "micro message"), 微信读书 (WeChat Reading), 豆瓣 (Douban, "bean petal"), and QQ.

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

  • webcrawler
  • scrapy
  • crawlab
  • spiders-management

TranslatedPython open-source project "The Road to Self-Taught Programming", with step-by-step tutorials for beginners: AI Lab, treasure videos, data structures, study guides, hands-on machine learning, hands-on deep learning, web crawlers, big-company interview experiences, programmer life, resource sharing.

TranslatedLive stream recording software that supports looped monitoring and recording multiple streamers at once. It supports recording live streams from 40+ platforms, including Douyin, TikTok, Youtube, Kuaishou, Huya, Douyu, Bilibili, Xiaohongshu, pandatv, sooplive, flextv, popkontv, twitcasting, winktv, Baidu, Weibo, Kugou, 17Live, Twitch, Acfun, CHZZK, shopee, and more

  • douyin-live
  • douyin-api
  • douyulive
  • live-recorder

TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database

  • javbus
  • avmoo
  • javlibrary
  • spider

TranslatedXiaohongshu (Little Red Book) crawler data collection, Xiaohongshu reverse engineering, private messages, live streaming, full-domain operations solution for Xiaohongshu

TranslatedINFO-SPIDER is a crawler toolbox 🧰 that brings many data sources together in one place, designed to help users get their own data back safely and quickly. The tool's code is open source and the process is transparent. Supported data sources include GitHub, QQ Mail, NetEase Mail, Alibaba Mail, Sina Mail, Hotmail, Outlook Mail, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ groups, generating a Moments (WeChat friend circle) photo album, browser browsing history, 12306, cnblogs (博客园), CSDN blog, OSChina blog, Jianshu.

TranslatedMore and more websites have anti-crawler features: some hide key data in images, and some use inhuman CAPTCHAs. This is a repository of anti-anti-crawler code, built to improve skills by contending with websites of different characteristics (with no malicious intent). (Submissions of websites that are hard to scrape are welcome.) (The project is paused due to work reasons.)

TranslatedProvince, city, district/county and township level (three-level or four-level) city data, with pinyin annotations, coordinates, and administrative region boundary ranges; latest collection on April 3, 2026; provides csv format files, supports online conversion into multi-level linked js code and general json format, provides software to convert into shp, geojson, sql and import into databases; includes js collection source code that runs in the browser, combining administrative division data from the Ministry of Civil Affairs of the People's Republic of China, China National Geographical Names Information Library, the Bureau of Statistics, Amap (Gaode Map), and Tencent Maps