Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
Repository Index
47,133 repositories indexed
49–72 of 86 · Most stars
TranslatedGive your AI a body, and a bit of human touch — a QQ companion chatbot that remembers things and reads the occasion, plus an MCP server that can control Android phones
Are you an early 🐤 or a night 🦉? Let's check out in gist
- pinned-gist
- awesome-pinned-gists
Functional HTML scraping and rewriting with CSS in OCaml
The unix-way web crawler
Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library
- golang
- unmarshall
- goquery
- selectors
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
- golang
- web-crawler
- crawling
- scraping
Playwright PHP library for browser automation: navigation, E2E tests, assertions, screenshots, and so much more!
- browser
- browser-automation
- php
- php-browser
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Self-hosted, invisible CAPTCHA that detects AI agents driving real browsers. GDPR, WCAG 2.2 AA, and EAA: no cookies, no third-party data sharing, and no visual or audio puzzle. With a false-positive budget enforced in CI. Proof of work, drop-in for Turnstile and reCAPTCHA.
- anti-bot
- behavioral-analysis
- bot-detection
- captcha
TranslatedAI subscription and card-shop price comparison platform: say what you need and let AI help you choose. Aggregates top-up service or account offers for ChatGPT / Codex, Claude, Gemini and more, and compares price trends, top-up channels and warranties. AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.
- ai
- price-comparison
- self-hosted
- web-scraping
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
🐦💬 Update a gist to contain your latest tweet
- pinned-gist
Translated🎮 Update profile README / pinned gist to contain your Steam playtime leaderboard.
- steam
- steam-playtime-leaderboard
- pinned-gist
- profile
Translated📊 Update profile README / pinned gist to contain your weekly WakaTime stats. This is a Golang implementation, see the original version https://github.com/matchai/waka-box. Display your weekly WakaTime language usage statistics in your profile README / pinned gist.
- wakatime-api
- wakatime
- pinned-gist
- weekly-wakatime-stats
📌 Update a gist to contain your GitHub stats
- gist
- github-actions
- github
- pinned-gist
Automate the process of shipping Web Extensions for Chrome, Firefox, Safari, Opera and Edge
Ping pong, chess, pool and other games bot for Slack.
An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.
Search the web and primary sources, read full pages, and check every citation. MCP server for Claude, Cursor, Codex and more. Runs on your machine.
Super simple deployment tool
- deployment
- dropship
- hcl
- shipping
Practical pinned gist which show your latest medium status 📌
- awesome-pinned-gists
- medium-stats
- github-gists
- github-action
Website to Scrapping all writeup from http://ctftime.org/ and you can organize which to read first!