Open Source Web Crawler for Java
Repository Index
72,862 visible repositories in the index
73–96 of 227 · Most stars
TranslatedPython web crawler tutorial that takes you from zero to one, covering JS reverse engineering, Selenium, Tesseract OCR recognition, the use of MongoDB, and the Scrapy framework
A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.
- bittorrent
- dht
- prowlarr
- radarr
Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.
- comic-downloader
- novel-downloader
- cejs
- downloader
The Ultimate Information Gathering Toolkit
- cms-detection
- directory-finder
- dns-lookup
- information-gathering
A list of AI agents and robots to block.
Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes
- crawler
- proxypool
Eases DOM navigation for HTML and XML documents
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
- crawler
- python3
- scylla
- proxy-pool
Headless Chrome .NET API
- puppeteer
- chrome
- chromium
- automation
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
- scanner
- crawler
- information-gathering
- admin-scanner
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- endpoints
- endpoint-discovery
- bugbounty
- crawler
新闻网页正文通用抽取器 Beta 版.
- python3
- webcrawler
- webspider
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
Scrape Google AI Mode responses without blocks on a large scale.
- ai-mode
- google-ai
- google-ai-mode
- proxy-scrape
A curated list of awesome ETL frameworks, libraries, and software.
Every web site provides APIs.
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
- aitools
- crawl
- scrape
- serp
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
- scrapy
- distributed
- webspider
- scrapyd
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
- scrapy
- python
- crawl
- crawler
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
- java
- webcrawling
- warc
- heritrix