Repository Index

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

• web-scraping
• text-extraction
• nlp
• html2text

Mechanize is a ruby library that makes automated web interaction easy.

• scraping
• web
• ruby

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

• crawler
• dotnetcore
• cross-platform
• csharp

A Python module to bypass Cloudflare's anti-bot page.

• cloudflare
• anti-bot-page
• protected-page
• scrape

A browser testing and web crawling library for PHP and Symfony

• scraping
• e2e-testing
• webdriver
• selenium

Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.

• datadome
• perimeterx
• akamai
• antidetect-browser

Get info from any web service or page

• opengraph
• twitter-cards
• embeds
• scraping

i'm typing when you're typing

Async Python 3.6+ web scraping micro-framework based on asyncio

• asyncio
• aiohttp
• asyncio-spider
• crawler

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

• fredy
• immowelt
• appartment
• appartments

A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)

Simple web scraping for R

• r
• web-scraping
• html

A chat bot for Slack (https://slack.com).

:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine

• meta-search-engine
• rust
• search-engine
• fast

Crawly, a high-level web crawling & scraping framework for Elixir.

• elixir
• erlang
• scraper
• scraping

Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

• xquery
• xml
• html
• json

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

• spider
• ruby
• spider-links
• crawler

:scissors: High performance, multi-threaded image scraper

• python
• pypi
• scraper
• scraping

A AI Browser run any site JavaScript from Python with the real Firefox engine. Headless SpiderMonkey + Necko for automation and scraping.

• anti-bot
• automation
• firefox
• headless-browser

Extract structured data from web sites. Web sites scraping.

• golang
• golang-library
• extract-data
• chrome-fetcher

TranslatedGive your AI a body, and a bit of human touch — a QQ companion chatbot that remembers things and reads the occasion, plus an MCP server that can control Android phones

• adb
• agent
• android
• chatbot

Functional HTML scraping and rewriting with CSS in OCaml

• scraping
• html
• soup
• ocaml

The unix-way web crawler

• cli
• crawler
• golang-application
• unix-way