Sign in

Repository Index

108,094 visible repositories in the index

Tags: Automation, bots, and scraping
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters (1)

Filters

Clear all filters

1–24 of 55 · Most stars

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

  • webcrawler
  • scrapy
  • crawlab
  • spiders-management

bridge between mattermost, IRC, gitter, xmpp, slack, discord, telegram, rocketchat, twitch, ssh-chat, zulip, whatsapp, keybase, matrix, microsoft teams, nextcloud, mumble, vk and more with REST API (mattermost not required!)

TranslatedAn API framework with a modular design based on Gin, which wraps commonly used features, is simple to use, and is dedicated to fast business development. For example, it supports cors cross-origin requests, jwt signature verification, zap log collection, panic exception capture, trace link tracing, prometheus monitoring metrics, swagger document generation, viper configuration file parsing, the gorm database component, the gormgen code generation tool, the graphql query language, errno unified definition of error codes, the use of gRPC, cron scheduled tasks, and so on.

  • golang
  • go
  • api-gin
  • gin

Swiss-army tool for scraping and extracting data from online assets, made for hackers

  • css
  • curl
  • gjson
  • json

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

  • endpoints
  • endpoint-discovery
  • bugbounty
  • crawler

GoClaw - GoClaw is OpenClaw rebuilt in Go — with multi-tenant isolation, 5-layer security, and native concurrency. Deploy AI agent teams at scale without compromising on safety.

Shadow any website for offline viewing, with the JavaScript stripped out

Save Any Telegram File to Anywhere 📂 (Alist, Disk, Webdav, S3, Rclone...) . Support restrict saving content and files beyond telegram.

Point it at any web page and it finds the video, transcodes it, and streams it to your TV in real time. It even burns in subtitles.

An app that helps you monitor your Kubernetes cluster, debug critical deployments & gives recommendations for standard practices

  • kubernetes
  • kubernetes-monitoring
  • kubernetes-controller
  • chatops

An archiving tool with an IM-style interface that prioritizes privacy and accessibility, integrated with various archival services including Internet Archive, archive.today, Ghostarchive, IPFS, Telegraph, and file systems.

  • telegram
  • archive
  • wayback-machine
  • save-the-internet