Repository Index

Scrapy+Splash for JavaScript integration

• scrapy
• headless-browsers

TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok

• automation
• content-monitoring
• crawler
• douyin

🤖 Top-rated tools to scrape all major public sections from Facebook, Instagram, and Twitter (X) including posts (likes/comments), photos/videos, contact information, followers, following and much more.

• facebook-page-insights
• instagram-followers-scraper
• facebook-ads-metrics
• facebook-contact-info

A service daemon to run Scrapy spiders

A browser testing and web crawling library for PHP and Symfony

• scraping
• e2e-testing
• webdriver
• selenium

Make your job hunt easy by automating your application process with this Auto Applier

• python
• selenium
• webscraping
• automation

Perplexity Scraper Track brand mentions, analyze rankings, and gain competitor intelligence from Perplexity. Get started in minutes.

• ai-scraper
• llm-api
• llm-scraper
• llm-scraping

Node.js scraper to get data from Google Play

• google-play
• scraper
• crawler
• api

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

• twitter
• scraper
• twitter-api
• twitter-bot

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

• search-engine
• python
• scraping
• crawler

Foundational low latency web data collecting in Rust

• crawler
• rust
• spider
• headless-chrome

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

• go
• scraping
• scraper
• crawler

admin ui for scrapy/open source scrapinghub

• scrapy
• dashboard
• scrapyd
• scrapy-ui

Scrape metadata from any URL using Open Graph, JSON-LD, HTML meta tags, and smart fallbacks.

• metadata
• article
• cheerio
• extract

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

• llm
• mcp
• modelcontextprotocol
• scraping

Generate and download e-books from online sources.

• lightnovel
• termux
• web-scraper
• console-app

日本电影元数据刮削器，配合kodi,emby,plex等本地媒体管理工具使用。可批量抓取，也可单个抓取。可抓取子目录下视频，多集视频（-cd1/-cd2）,带字幕作品（-c., -C.）。批量添加emby演员头像。

• jav
• jav-scraper
• emby
• kodi

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

• querylist
• crawler
• spider
• scraper

Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva, reCAPTCHA, ThreatMetrix, Adscore

• akamai
• cloudflare
• incapsula
• perimeterx

Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.

• fingerprinting
• playwright
• puppeteer
• scraping

🔥 Visual workflow builder for AI agents powered by Firecrawl - drag-and-drop web scraping pipelines with real-time execution

A cloudflare verification bypass script for webscraping

• cloudflare-bypass
• bypass
• cloudflare
• cloudflare-bypass-script

Downloads and archives content from reddit

• reddit
• imgur
• gfycat
• scraper

A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript

• playwright
• playwright-python
• scraping
• screenshot-utility