Redis-based components for Scrapy.
- scrapy
- crawler
- distributed
- redis
142,187 visible repositories in the index
1–17 of 17 · Most stars
Redis-based components for Scrapy.
Better GitHub statistics images for your profile, with stats from private repos too
A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.
Make a ZIM file from any Web site and surf offline!
A command-line tool to get valuable information out of AWS CloudTrail
Undetected NodeJS version of the Playwright testing and automation library.
豆瓣电影/豆瓣读书 Scarpy 爬虫
This library was created for testing and training purposes to retrieve the page source of websites, create Cloudflare Turnstile tokens and create Cloudflare WAF sessions.
🤖 AI-powered web scraping editor with visual workflow builder. Build, test & deploy web scrapers using natural language. Powered by ScrapeGraphAI & LangGraph.
Two dumb distributed crawlers
100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.
狠心开源企业级舆情新闻爬虫项目:支持任意数量爬虫一键运行、爬虫定时任务、爬虫批量删除;爬虫一键部署;爬虫监控可视化; 配置集群爬虫分配策略;👉 现成的docker一键部署文档已为大家踩坑 an enterprise-grade public opinion news crawler project: supports one-click operation of any number of crawlers, scheduled crawler tasks, batch deletion of crawlers; one-click deployment of crawlers
OSINT tool - gets data from services like shodan, censys etc. in one app
The simplest demo of chrome automation by python and selenium in AWS Lambda
Free Web Scraping Tool with Java
Proxy server to bypass Cloudflare protection.
Self-hosted manga collection manager with automatic metadata, MyAnimeList import, Docker support, and detailed collection statistics.