Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 35 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Open-source live-chat, email support, omni-channel desk. An alternative to Intercom, Zendesk, Salesforce Service Cloud etc. 🔥💬
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
📷 Instagram Bot - Tool for automated Instagram interactions
- clarifai
- automation
- script
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
CLI for common Playwright actions. Record and generate Playwright code, inspect selectors and take screenshots.
The repository host gives no description.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
An NLP library for building bots, with entity extraction, sentiment analysis, automatic language identify, and so more
Mechanize is a ruby library that makes automated web interaction easy.
- scraping
- web
- ruby
Network Automation and Programmability Abstraction Layer with Multivendor support
A GitHub Action invoking the Gemini CLI.
Lightweight Node.js library for building Slack Block Kit UIs, with a declarative syntax inspired by SwiftUI.
Steam's protocol in Go to allow automation of different actions on the Steam network without running an actual Steam client. Includes APIs for friends, chatting, trading, trade offers and TF2 crafting.
- steam
- steamkit
Configuration as Code, use YAML to write automated workflows that run on Deno, with any Deno modules, Typescript/Javascript codes
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Predictable JavaScript abstractions for Discord API.
The Frank!Framework is an easy-to-use, stateless integration framework which allows (transactional) messages to be modified and exchanged between different systems.
- java
- system-integration
- open-source
- community-driven
📌 Update a gist to contain your GitHub stats
- gist
- github-actions
- github
- pinned-gist
A robust web archive analytics toolkit
- python
- web
- warc
- bigdata