Sign in

Repository Index

142,273 visible repositories in the index

Tags: Web scraper
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 68 · Most stars

TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

  • webcrawler
  • scrapy
  • crawlab
  • spiders-management

Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.

Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha

Swiss-army tool for scraping and extracting data from online assets, made for hackers

  • css
  • curl
  • gjson
  • json

An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

  • crawler
  • dotnetcore
  • cross-platform
  • csharp

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

  • scanner
  • crawler
  • information-gathering
  • admin-scanner

Use Google News Scraper API to obtain the latest global news for your project, including a wide range of sources, headlines, URLs, and publication dates from the Google News platform.

  • google-news-scraper
  • google-scraper
  • google-scraper-api
  • news-scraper

🤖 Top-rated tools to scrape all major public sections from Facebook, Instagram, and Twitter (X) including posts (likes/comments), photos/videos, contact information, followers, following and much more.

  • facebook-page-insights
  • instagram-followers-scraper
  • facebook-ads-metrics
  • facebook-contact-info

AutoCLI is a Blazing fast, memory-safe command-line tool — Fetch information from any website with a single command. Covers Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, and 55+ sites, with support for controlling Electron desktop apps, integrating local CLI tools (gh, docker, kubectl), now powered by AutoCLI.ai .

Point it at any web page and it finds the video, transcodes it, and streams it to your TV in real time. It even burns in subtitles.