Apache Nutch is an extensible and scalable web crawler
- java
- nutch
- web-crawler
- crawling
Repository Index
72,862 visible repositories in the index
97–120 of 227 · Most stars
Videodl: A lightweight video downloader written in pure python. (轻量级视频下载器，优先高清无水印，支持抖音，快手，小红书，B站，TikTok，YouTube，FIFA+，优酷，腾讯，爱奇艺，1905电影网，乐视，芒果，咪咕，PPTV，搜狐，Facebook，Twitter，新浪微博，今日头条，网易公开课，全民K歌，CCTV央视频，酷狗音乐MV，新片场，知乎，百度贴吧，TED等海量流媒体平台)
- video-downloader
- cntv
- douyin
- crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
- automation
- content-monitoring
- crawler
- douyin
visualized crawler & ETL IDE written with C#/WPF
Article publishing platform that automatically distributes your articles to various media channels
Dungeon Crawl: Stone Soup official repository
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
A browser testing and web crawling library for PHP and Symfony
All In One Web Recon
- python3
- pentesting
- pentest-tool
- pentesting-tools
Automatically crawl arXiv papers daily and summarize them using AI. Illustrating them using GitHub Pages.
A powerful browser crawler for web vulnerability scanners
- headless-chrome
- crawler
- chrome-devtools
- golang
Gospider - Fast web spider written in Go
Node.js scraper to get data from Google Play
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
- scraper
- twitter-api
- twitter-bot
Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.
- google-trend
- google-trend-api
- google-trend-strategy
owllook-小说搜索引擎
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
- search-engine
- python
- scraping
- crawler
DecryptLogin: APIs for loginning some websites by using requests.
- requests
- login
- python3
- spider
Foundational low latency web data collecting in Rust
https://spatie.be/docs/crawler
- php
- crawler
- guzzle
- concurrency
Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal, GhostArchive & Intelligence X!
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.