DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
Repository Index
72,862 visible repositories in the index
49–72 of 135 · Most stars
Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes
- crawler
- proxypool
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
- crawler
- python3
- scylla
- proxy-pool
Headless Chrome .NET API
- puppeteer
- chrome
- chromium
- automation
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
- scanner
- crawler
- information-gathering
- admin-scanner
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- endpoints
- endpoint-discovery
- bugbounty
- crawler
新闻网页正文通用抽取器 Beta 版.
- python3
- webcrawler
- webspider
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
Every web site provides APIs.
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
- aitools
- crawl
- scrape
- serp
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
- scrapy
- distributed
- webspider
- scrapyd
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
- java
- webcrawling
- warc
- heritrix
Apache Nutch is an extensible and scalable web crawler
- java
- nutch
- web-crawler
- crawling
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
- automation
- content-monitoring
- crawler
- douyin
visualized crawler & ETL IDE written with C#/WPF
Article publishing platform that automatically distributes your articles to various media channels
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
A browser testing and web crawling library for PHP and Symfony
Automatically crawl arXiv papers daily and summarize them using AI. Illustrating them using GitHub Pages.
A powerful browser crawler for web vulnerability scanners
- headless-chrome
- crawler
- chrome-devtools
- golang
Gospider - Fast web spider written in Go
Node.js scraper to get data from Google Play