Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 43 · Most stars
TranslatedA visual no-code/code-free web crawler/spider. 易采集 (EasySpider, "easy collection"): a visual browser automation testing/data collection/web crawler software that lets you design and execute crawler tasks graphically without code. Alias: ServiceWrapper, an intelligent service wrapping system for Web applications.
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- wget
- browser-bookmarks
- pinboard
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
A little like that j-thing, only in Go.
- selector-strings
- jquery
- html-parsing
- goquery
Pythonic HTML Parsing for Humans™
TranslatedAn online tool for batch downloading WeChat Official Account articles. It supports exporting read count and comment data, requires no environment setup, and can be used through an online website. It supports private deployment with docker and deployment on Cloudflare. It supports downloading various file formats, and the HTML format can restore article layout and styles 100%.
- download
- wechat-article
- wechat-download
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
A graph node engine and editor written in Javascript similar to PD or UDK Blueprints, comes with its own editor in HTML5 Canvas2D. The engine can run client side or server side using Node. It allows to export graphs as JSONs to be included in applications independently.
- graph
- modular
- javascript
- visual-programming
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
The next web scraper. See through the <html> noise.
SwiftSoup: Pure Swift HTML Parser, with best of DOM, CSS, and jquery (Supports Linux, iOS, Mac, tvOS, watchOS)
- swift
- swiftsoup
- parse
- html-document
Twitter Text Libraries. This code is used at Twitter to tokenize and parse text to meet the expectations for what can be used on the platform.
- twitter-text
- tweet
- unicode
Generate critical css for your web pages
PufferPanel is an open source game server management panel, designed for both small networks and personal use
- game-server
- minecraft
- srcds
- panel
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
Simple web scraping for R
Visually build a full featured chat bot for Telegram, Facebook Messenger, Whatsapp and Slack with Node-RED. Almost no coding skills required.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
- anti-crawling
- cloud-native
- crawler
- deception
Serenity BDD is a test automation library designed to make writing automated acceptance tests easier, and more fun.
High-performance HTML5 parser for Ruby based on Lexbor, with support for both CSS selectors and XPath.
Functional HTML scraping and rewriting with CSS in OCaml