使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Repository Index
142,273 visible repositories in the index
97–120 of 452 · Most stars
Scrapy+Splash for JavaScript integration
- scrapy
- headless-browsers
🤖 Top-rated tools to scrape all major public sections from Facebook, Instagram, and Twitter (X) including posts (likes/comments), photos/videos, contact information, followers, following and much more.
- facebook-page-insights
- instagram-followers-scraper
- facebook-ads-metrics
- facebook-contact-info
A service daemon to run Scrapy spiders
A browser testing and web crawling library for PHP and Symfony
Make your job hunt easy by automating your application process with this Auto Applier
- python
- selenium
- webscraping
- automation
Perplexity Scraper Track brand mentions, analyze rankings, and gain competitor intelligence from Perplexity. Get started in minutes.
- ai-scraper
- llm-api
- llm-scraper
- llm-scraping
Node.js scraper to get data from Google Play
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
- scraper
- twitter-api
- twitter-bot
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
- search-engine
- python
- scraping
- crawler
Foundational low latency web data collecting in Rust
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
admin ui for scrapy/open source scrapinghub
- scrapy
- dashboard
- scrapyd
- scrapy-ui
Scrape metadata from any URL using Open Graph, JSON-LD, HTML meta tags, and smart fallbacks.
- metadata
- article
- cheerio
- extract
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Generate and download e-books from online sources.
- lightnovel
- termux
- web-scraper
- console-app
日本电影元数据刮削器,配合kodi,emby,plex等本地媒体管理工具使用。可批量抓取,也可单个抓取。可抓取子目录下视频,多集视频(-cd1/-cd2),带字幕作品(-c., -C.)。批量添加emby演员头像。
- jav
- jav-scraper
- emby
- kodi
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.
- fingerprinting
- playwright
- puppeteer
- scraping
Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva, reCAPTCHA, ThreatMetrix, Adscore
- akamai
- cloudflare
- incapsula
- perimeterx
🔥 Visual workflow builder for AI agents powered by Firecrawl - drag-and-drop web scraping pipelines with real-time execution
A cloudflare verification bypass script for webscraping
- cloudflare-bypass
- bypass
- cloudflare
- cloudflare-bypass-script
Downloads and archives content from reddit
- imgur
- gfycat
- scraper
invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
- datadome
- perimeterx
- akamai
- antidetect-browser