:scissors: High performance, multi-threaded image scraper
Repository Index
47,133 repositories indexed
73–96 of 127 · Most stars
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
- automation
- ecommerce
- filamentphp
- laravel
A AI Browser run any site JavaScript from Python with the real Firefox engine. Headless SpiderMonkey + Necko for automation and scraping.
- anti-bot
- automation
- firefox
- headless-browser
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
The unofficial HLTV Node.js API
Translated✈️ Self-hosted multi-channel airfare price monitoring: flight detail collection → DingTalk rich-media push → four-layer strong alerts when the target price is met. Domestic routes and international routes (destinations on five continents) share the same collect-judge-push pipeline; data stays on your own machine, all configuration is done on a web page, open source and free.
- crawler
- dingtalk
- flight
- playwright
Steam's protocol in Go to allow automation of different actions on the Steam network without running an actual Steam client. Includes APIs for friends, chatting, trading, trade offers and TF2 crafting.
- steam
- steamkit
Xiryn — the open-source personal agent. Does everything Muse, Grok Bot and Dots do, on your own computer: each agent gets its own Linux desktop in Docker, any model with your keys, an encrypted vault, and control from Telegram.
Functional HTML scraping and rewriting with CSS in OCaml
WarcDB: Web crawl data as SQLite databases.
TranslatedAn Agent Skill for scraping the complete comment section of a single Douyin video (top-level comments + nested replies): it copies the page's real requests inside a logged-in Chrome and leaves the signing to the page's SDK
- agent-skill
- claude-code
- comments-scraper
- douyin
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
The unix-way web crawler
Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
DocSearch - Scraper
- algolia
- docsearch
- documentation
- scraper
A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library
- golang
- unmarshall
- goquery
- selectors
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
- golang
- web-crawler
- crawling
- scraping
Playwright PHP library for browser automation: navigation, E2E tests, assertions, screenshots, and so much more!
- browser
- browser-automation
- php
- php-browser
Mac app for website auditing and crawling
- seo
- website-auditing
- swift
- mac
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Self-hosted, invisible CAPTCHA that detects AI agents driving real browsers. GDPR, WCAG 2.2 AA, and EAA: no cookies, no third-party data sharing, and no visual or audio puzzle. With a false-positive budget enforced in CI. Proof of work, drop-in for Turnstile and reCAPTCHA.
- anti-bot
- behavioral-analysis
- bot-detection
- captcha
A modern async client library for Rust, supports Slack Web / Events API/Socket Mode and Block Kit.
- slack-api
- slack
- rust
- rust-crate
TranslatedAI subscription and card-shop price comparison platform: say what you need and let AI help you choose. Aggregates top-up service or account offers for ChatGPT / Codex, Claude, Gemini and more, and compares price trends, top-up channels and warranties. AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.
- ai
- price-comparison
- self-hosted
- web-scraping
:vertical_traffic_light: Block malicious crawlers with Cloudflare Firewall Rules
- cloudflare
- firewall
- user-agent
- firewall-rules