Repository Index

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

Make optimisation of images part of your automated build process

• imageoptim
• image-optimisation
• jpegmini
• pngquant

The open source and enterprise-ready webhooks service 🦀

• webhooks
• rust
• webhook-server
• webhook-service

🤖 Telegram Bot API PHP SDK. Lets you build Telegram Bots easily! Supports Laravel out of the box.

• telegram
• telegram-bot
• telegram-bot-sdk
• php

Generate critical css for your web pages

• performance
• css
• web
• nodejs

Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

• fredy
• immowelt
• appartment
• appartments

Strategies to Gekko trading bot with backtests results and some useful tools.

• gekko
• strategies
• gekko-strategies
• gekko-backtest

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• ruby
• scraper
• crawler
• dsl

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

• java
• elasticsearch
• full-text-search
• lucene

Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

• xquery
• xml
• html
• json

:scissors: High performance, multi-threaded image scraper

• python
• pypi
• scraper
• scraping

Extracts information about web pages, like youtube videos, twitter statuses or blog articles.

• php
• embed
• oembed
• opengraph

Erik is an headless browser based on WebKit. An headless browser allow to run functional tests, to access and manipulate webpages using javascript.

• webkit
• headless-browsers
• erik
• swift

Slack Developer Kit (including Bolt for Java) for any JVM language

• java
• slack
• slack-api
• slack-bot

Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.

Zotero extension that combats link rot by archiving webpages and journal articles.

An R package for sending messages from R to Slack

• r
• slack
• rstats

One-link downloader for the Mac (videos, documents, images, articles, web pages), with optional automatic YouTube monitoring that translates and subtitles new videos.

• downloader
• google-apps-script
• macos
• subtitles

Slack integration for Django, using the templating engine to generate messages

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

• ai-agents
• browser-automation
• data-extraction
• firecrawl

Extract URLs to stylesheets, scripts, links, images or HTML imports from HTML

Messages to slack from command line

• python
• slack
• command-line