Mac app for website auditing and crawling
- seo
- website-auditing
- swift
- mac
Repository Index
47,133 repositories indexed
97–120 of 127 · Most stars
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
TranslatedAI subscription and card-shop price comparison platform: say what you need and let AI help you choose. Aggregates top-up service or account offers for ChatGPT / Codex, Claude, Gemini and more, and compares price trends, top-up channels and warranties. AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.
- ai
- price-comparison
- self-hosted
- web-scraping
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
Extract URLs to stylesheets, scripts, links, images or HTML imports from HTML
Extract web archive data using Wayback Machine and Common Crawl
- golang
- crawler
- commoncrawl
- wayback-machine
Web spider as a service, spider on serverless
- web-spider
- serverless
- javascript
- api
Go implementation of noisy HTTP/DNS traffic generator
A robust web archive analytics toolkit
- python
- web
- warc
- bigdata
Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app
Parse sample tests of Codeforces competitions, and generate tests automatically.
XPath/XQuery 3.1 interpreter for Pascal with compatibility modes for XPath 2.0/XQuery 1.0/3.0, custom and JSONiq extensions, pattern matching, XML/HTML/JSON parsers and classes for HTTP/S requests
- xpath
- xquery
- pascal
- html
Python/FastAPI application for monitoring Brazilian public-sector IT opportunities, with automated data collection, job filters, and official-source tracking.
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
- go
- golang
- goquery
- colly
An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.
Credential Phish Analysis and Automation
Scrapy project for data capture of vgchartz
Search the web and primary sources, read full pages, and check every citation. MCP server for Claude, Cursor, Codex and more. Runs on your machine.
api for liquipedia.net
- dota2
- liquipedia
- python3
- counter-strike
Webpage summary extractor using Facebook Open Graph and arc90's readability
- readability
- scraper
- opengraph
Create HTML snippets/embeds from URLs using info from oEmbed, Open Graph, meta tags.
Get user info of an npm user
Download Medium posts as clean Markdown, preserving structure, images, links, code blocks, and common embeds for plain Markdown or Jekyll workflows.
- ruby
- medium
- medium-articles
- markdown
Code to download, process, and analyse the Online Encyclopedia of Integer Sequences