Sign in

Repository Index

142,187 visible repositories in the index

Tags: Automation, bots, and scraping
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters (1)

Filters

Clear all filters

1–24 of 35 · Most stars

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

  • webcrawler
  • scrapy
  • crawlab
  • spiders-management

A Chrome DevTools Protocol driver for web automation and scraping.

  • cdp
  • chrome-headless
  • chrome-devtools
  • chrome-devtools-protocol

Swiss-army tool for scraping and extracting data from online assets, made for hackers

  • css
  • curl
  • gjson
  • json

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

  • endpoints
  • endpoint-discovery
  • bugbounty
  • crawler

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Self-hosted web search for AI agents and SEO. Google, Bing, Yandex, Baidu, DuckDuckGo & Ecosia, rendered in a real browser: ranked results and clean page Markdown. API, CLI, MCP 🎉

  • baidu
  • google
  • serp
  • serpapi

Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic level (JA3/JA4, Akamai fingerprint, header order). Supports HTTP/1.1, HTTP/2, HTTP/3, sessions, cookies, and proxies.

  • anti-bot
  • bot-detection
  • browser-fingerprint
  • browser-fingerprinting

开箱即用的本地 JAV/视频 刮削、管理、播放软件,支持命令行一键安装和 Docker 部署。只需简单添加目录,即可打造你的私人 JAV/视频 媒体库,带给你顶级的浏览体验,懒人必备。| Your local JAV/video manager.

  • jav
  • porn
  • javbus
  • javlibrary

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

Extract structured data from web sites. Web sites scraping.

  • golang
  • golang-library
  • extract-data
  • chrome-fetcher