TranslatedA visual no-code/code-free web crawler/spider. 易采集 (EasySpider, "easy collection"): a visual browser automation testing/data collection/web crawler software that lets you design and execute crawler tasks graphically without code. Alias: ServiceWrapper, an intelligent service wrapping system for Web applications.
Repository Index
47,133 repositories indexed
1–24 of 89 · Most stars
Elegant Scraper and Crawler Framework for Golang
SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
- footprinting
- osint
- threatintel
- python
Python ProxyPool for web spider
TranslatedA Python web scraping tutorial series, learning Python web scraping from 0 to 1, including browser packet capture and mobile APP packet capture with tools such as fiddler and mitmproxy, the use of the various modules involved in web scraping, such as requests, beautifulSoup, selenium, appium, scrapy, etc., as well as IP proxies, CAPTCHA recognition, using Mysql and MongoDB databases with Python, multi-threaded and multi-process scraping, reverse engineering and cracking of CSS scraping encryption, JS scraping reverse engineering, distributed scraping, practical scraping project examples, etc.
- python3
- python-script
- python-spider
Translated:rainbow:Python3 web crawler practice: Taobao, JD.com, NetEase Cloud Music, Bilibili, 12306, Douyin, Biquge, comic and novel downloads, music and movie downloads, etc.
- python-spider
- python3
- python
- webspider
A next-generation crawling and spidering framework.
- crawler
- web-spider
- gocrawler
- spider-framework
Translated😮Python simulated login for some large websites, plus some simple web crawlers. I hope they are helpful to you❤️. If you like it, remember to give it a star🌟
- python
- spiders
- selenium
- twitter-bot
TranslatedSome interesting examples of python crawlers that are friendly to beginners. Mainly crawls websites such as 淘宝 (Taobao, "searching for treasure"), 天猫 (Tmall, "sky cat"), 微信 (WeChat, "micro message"), 微信读书 (WeChat Reading), 豆瓣 (Douban, "bean petal"), and QQ.
Incredibly fast crawler designed for OSINT.
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
TranslatedPython open-source project "The Road to Self-Taught Programming", with step-by-step tutorials for beginners: AI Lab, treasure videos, data structures, study guides, hands-on machine learning, hands-on deep learning, web crawlers, big-company interview experiences, programmer life, resource sharing.
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
TranslatedLive stream recording software that supports looped monitoring and recording multiple streamers at once. It supports recording live streams from 40+ platforms, including Douyin, TikTok, Youtube, Kuaishou, Huya, Douyu, Bilibili, Xiaohongshu, pandatv, sooplive, flextv, popkontv, twitcasting, winktv, Baidu, Weibo, Kugou, 17Live, Twitch, Acfun, CHZZK, shopee, and more
- douyin-live
- douyin-api
- douyulive
- live-recorder
TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
- javbus
- avmoo
- javlibrary
- spider
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
:spider_web: input/output for many mesh formats
- vtk
- pypi
- python
- mesh
The repository host gives no description.
Async Python 3.6+ web scraping micro-framework based on asyncio
- asyncio
- aiohttp
- asyncio-spider
- crawler
Anemone web-spider framework
:cloud: The SCSS framework for the modern web.
- css-framework
- css
- web-development
- stylesheet