zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
Repository Index
132,241 visible repositories in the index
169–192 of 248 · Most stars
ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.
- chatgpt
- embedding
- gpt-35-turbo
- news-extractor
一个超级轻量的百度图片爬虫
AI-powered StartUp Accelerator Engine built with Next.js, LangChain, PostgreSQL + pgvector. Upload, organize, and chat with documents. Includes predictive missing-document detection, role-based workflows, and page-level insight extraction.
LAYRA—an enterprise-ready, out-of-the-box solution—unlocks next-generation intelligent systems powered by visual RAG and limitless visual multi-step agent workflow orchestration.
- agent
- document-parser
- knowledge-base
- visual-rag
HTTP API for Scrapy spiders
- python
- crawling
- crawler
- scrapy
Crawler (Bot) searching for leaks on paste sites.
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
- skrape
- dom
- parse
- kotlin
Crawl BookCorpus
ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux
- download
- download-manager
- qt
- torrent-downloader
🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.
- ai
- ai-scraping
- data
- examples
tailwindui-crawler downloads the component HTML files locally
Social media (Weibo) comments analyzing toolbox in Chinese 微博评论分析工具, 实现功能: 1.微博评论数据爬取; 2.分词与关键词提取; 3.词云与词频统计; 4.情感分析; 5.主题聚类
- sentiment-analysis
- word-clouds
- lda
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
SEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...
- seo
- seo-tools
- internal-links
- headers
Simple but useful Python web scraping tutorial code.
- beautifulsoup
- scrapy
- regex
- urllib
brozzler - distributed browser-based web crawler
dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
- web-scraping
- man-in-the-middle
- proxy-server
- mitm
输入域名>爆破子域名>扫描子域名端口>发现扫描web服务>集成报告的全流程全自动扫描器。集成oneforall、masscan、nmap、dirsearch、crawlergo、xray等工具,另支持cdn识别、网页截图、站点定位;动态识别域名并添加功能、工具超时中断等
- crawlergo
- dirsearch
- nmap-http
- masscan
Doujinshi downloader 绅士漫画下载
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
- bot-detection
- browser-automation
- chromium
- fingerprinting
Java API For Chrome and Firefox
🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON.