Sign in

Repository Index

132,241 visible repositories in the index

Tags: Web scraper
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 45 · Most stars

A Chrome DevTools Protocol driver for web automation and scraping.

  • cdp
  • chrome-headless
  • chrome-devtools
  • chrome-devtools-protocol

TranslatedZotero translators for Chinese web page scraping 🎉 This is Zotero translators for Chinese Sites(beta), not the official Zotero repo

  • zotero
  • translators
  • javascript
  • cnki

TranslatedToday's Hot List (今日热榜), an aggregator website that gets the hot headlines from major popular websites. Written in Go, it uses multiple goroutines to fetch information asynchronously and quickly. Preview: https://mo.fish

  • hot
  • golang
  • spider
  • today-s-hot-list

An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

  • endpoints
  • endpoint-discovery
  • bugbounty
  • crawler

Point it at any web page and it finds the video, transcodes it, and streams it to your TV in real time. It even burns in subtitles.

Retrieve author and publication information from Google Scholar in a friendly, Pythonic way without having to worry about CAPTCHAs!

  • scholar
  • googlescholar
  • scholarly-articles
  • scholarly-communications

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.