Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 70 · Most stars
TranslatedA visual no-code/code-free web crawler/spider. 易采集 (EasySpider, "easy collection"): a visual browser automation testing/data collection/web crawler software that lets you design and execute crawler tasks graphically without code. Alias: ServiceWrapper, an intelligent service wrapping system for Web applications.
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- wget
- browser-bookmarks
- pinboard
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
UVdesk Open Source Community Helpdesk is a comprehensive ticketing support system designed for everyone, offering robust features to streamline customer support and collaboration.
- ticketing-system
- symfony4
- backbonejs
- opensource
A little like that j-thing, only in Go.
- selector-strings
- jquery
- html-parsing
- goquery
TranslatedDaidai Didi (带带弟弟, "take little brother along") general-purpose captcha recognition OCR, PyPI version
Pythonic HTML Parsing for Humans™
TranslatedAn online tool for batch downloading WeChat Official Account articles. It supports exporting read count and comment data, requires no environment setup, and can be used through an online website. It supports private deployment with docker and deployment on Cloudflare. It supports downloading various file formats, and the HTML format can restore article layout and styles 100%.
- download
- wechat-article
- wechat-download
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
A graph node engine and editor written in Javascript similar to PD or UDK Blueprints, comes with its own editor in HTML5 Canvas2D. The engine can run client side or server side using Node. It allows to export graphs as JSONs to be included in applications independently.
- graph
- modular
- javascript
- visual-programming
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
The next web scraper. See through the <html> noise.
SwiftSoup: Pure Swift HTML Parser, with best of DOM, CSS, and jquery (Supports Linux, iOS, Mac, tvOS, watchOS)
- swift
- swiftsoup
- parse
- html-document
Twitter Text Libraries. This code is used at Twitter to tokenize and parse text to meet the expectations for what can be used on the platform.
- twitter-text
- tweet
- unicode
Track GitHub trending repositories in your favorite programming language by native GitHub notifications!
- github
- trending
- trending-repositories
Generate critical css for your web pages
OpenSheetMusicDisplay renders sheet music in MusicXML format in browsers and headless with NodeJS. Based on VexFlow, OSMD is brought to you by PhonicScore.com.
- music
- musicxml
- mxl
- sheetmusic
PufferPanel is an open source game server management panel, designed for both small networks and personal use
- game-server
- minecraft
- srcds
- panel
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
Simple web scraping for R
µTask is an automation engine that models and executes business processes declared in yaml. ✏️📋
- workflow-engine
- workflow-automation
- golang-application
- devops-tools
Terraform without the state file bottleneck
- automation
- cloud-management
- devops
- github-actions