Repository Index

The next web scraper. See through the <html> noise.

The All in One Framework to Build Undefeatable Scrapers

• anti-bot
• anti-detection
• cloudflare-bypass
• cloudflare-scrape

TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:

• python3
• crawler
• baidu-tieba
• taobao-spider

Redis-based components for Scrapy.

• scrapy
• crawler
• distributed
• redis

Distributed crawler powered by Headless Chrome

• headless-chrome
• puppeteer
• jquery
• crawler

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

• mcp
• agent
• ai
• ai-agent

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

• high-availability
• scrapy
• ipproxy
• distributed

A social networking service scraper in Python

• python
• scraper
• social-network
• social-media

TikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.

TranslatedAn AV metadata scraper that aggregates data from multiple sites

• python
• kodi
• emby
• jellyfin

Undetected version of the Playwright testing and automation library.

• automation
• bot
• bots
• botting

Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.

• ai
• claude-code-plugin
• cli
• css

Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha

Swiss-army tool for scraping and extracting data from online assets, made for hackers

• css
• curl
• gjson
• json

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

• python
• bot
• client
• python3

Jobs scraper library for LinkedIn, Indeed, Glassdoor, ZipRecruiter, Bayt, Naukri, BDJobs & HelloWork

• indeed
• linkedin
• ziprecruiter
• jobs-scraper

A library that scrapes Linkedin for user data

• linkedin
• linkedin-scraper
• scraper
• profile

An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.

• ai-agent
• automation
• data-extraction
• event-driven-architecture

Mechanize is a ruby library that makes automated web interaction easy.

• scraping
• web
• ruby

Datascraper that is currently adult focused - Updated regularly

• onlyfans
• datascraping
• scraper
• archive

Open-source MCP server for LinkedIn. Give Claude and any MCP-compatible AI agent access to profiles, companies, jobs, and messages.

• linkedin
• linkedin-scraper
• mcp
• mcp-server

Lightweight, scriptable browser as a service with an HTTP API

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

• crawler
• dotnetcore
• cross-platform
• csharp

持续维护的新浪微博采集工具🚀🚀🚀

• scrapy
• python
• weibo
• weibospider