Repository Index

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

• pocket
• wget
• browser-bookmarks
• pinboard

An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.

• ai-agent
• automation
• data-extraction
• event-driven-architecture

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

• endpoints
• endpoint-discovery
• bugbounty
• crawler

Use Google News Scraper API to obtain the latest global news for your project, including a wide range of sources, headlines, URLs, and publication dates from the Google News platform.

• google-news-scraper
• google-scraper
• google-scraper-api
• news-scraper

JSFinder is a tool for quickly extracting URLs and subdomains from JS files on a website.

Get video info from Dailymotion, Vimeo, Wistia, and YouTube URLs.

A Oembed consumer library, that gives you information about urls. It helps you replace urls to youtube or vimeo for example, with their html embed code. It has advanced features like offline support, responsive embeds and caching support.

• oembed
• embed-videos
• youtube
• twitter

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

• ai-agents
• browser-automation
• data-extraction
• firecrawl

Extract URLs to stylesheets, scripts, links, images or HTML imports from HTML

Create HTML snippets/embeds from URLs using info from oEmbed, Open Graph, meta tags.

Go-native browser automation via Chrome DevTools Protocol. No Node.js, no relay process, no external dependencies.

• browser-automation
• cdp
• chrome-devtools-protocol
• golang