Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
- crawler
- python3
- scylla
- proxy-pool
132,241 visible repositories in the index
49–72 of 248 · Most stars
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
Headless Chrome .NET API
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
新闻网页正文通用抽取器 Beta 版.
A list of practical knowledge-building projects.
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单,功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手 / 视频号 / TikTok / 哔哩哔哩
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
Every web site provides APIs.
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Apache Nutch is an extensible and scalable web crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
visualized crawler & ETL IDE written with C#/WPF
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
A powerful browser crawler for web vulnerability scanners
Gospider - Fast web spider written in Go
Node.js scraper to get data from Google Play
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
owllook-小说搜索引擎
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
Foundational low latency web data collecting in Rust