A web privacy measurement framework
Repository Index
132,241 visible repositories in the index
121–144 of 248 · Most stars
A Unix-style personal search engine and web crawler for your digital footprint.
- search
- poseidon
- webcrawler
- personal-search
🤖 Scrape data from HTML websites automatically by just providing examples
- scraping
- crawling
- html
- machine-learning
Get Instagram posts/profile/hashtag data without using Instagram API
- instagram-crawler
- python
- webdriver
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
A scalable frontier for web crawlers
📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.
- crawler
- python
- instapy
The fastest dork scanner written in Go.
- golang
- dork-scanner
- crawler
- dorking
Python爬虫,京东自动登录,在线抢购商品
基于金融-司法领域(兼有闲聊性质)的聊天机器人,其中的主要模块有信息抽取、NLU、NLG、知识图谱等,并且利用Django整合了前端展示,目前已经封装了nlp和kg的restful接口
- deep-learning
- chatbot-cn
- tenserflow-serving
- intent-detection
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
- link-checker
- typescript
- broken-links
- 404
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Beanbun 是用 PHP 编写的多进程网络爬虫框架,具有良好的开放性、高可扩展性,基于 Workerman。
👧 美女写真套图爬虫(二)
- crawler
- meizi
Run a high-fidelity browser-based web archiving crawler in a single Docker container
- crawler
- crawling
- wacz
- warc
📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.
- articles
- articles-data
- crawler
- datasets-preparation
🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string
- user-agent-analysis
- user-agent
- user-agent-parser
- crawlers
Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片,视频
- tumblr
- videos
- photos
- crawler
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
Browser Detection for Laravel by hisorange!
- laravel
- mobile-detection
- user-agent
- user-agent-parser
Scalable Python web scraping scripts for +80 popular domains
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
- desktop-app
- flask
- free
- open-source
Crawly, a high-level web crawling & scraping framework for Elixir.
Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...