Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 130 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
🧡 Everything is RSSible
AI Agent Assistant & development framework that integrates lots of IM platforms, LLMs, plugins and AI feature, and can be your openclaw alternative. ✨
Open-source live-chat, email support, omni-channel desk. An alternative to Intercom, Zendesk, Salesforce Service Cloud etc. 🔥💬
A browser automation framework and ecosystem.
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
Let's use AI to Earn!
- auto-publish
- douyin
- electron-app
- electron-react
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
Self-Hosting Guide. Learn all about locally hosting (on premises & private web servers) and managing software applications by yourself or your organization. Including Cloud, LLMs, WireGuard, Automation, Home Assistant, and Networking.
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Command-line program to download image galleries and collections from several image hosting sites
- gallery
- downloader
- pixiv
- danbooru
UVdesk Open Source Community Helpdesk is a comprehensive ticketing support system designed for everyone, offering robust features to streamline customer support and collaboration.
- ticketing-system
- symfony4
- backbonejs
- opensource
TranslatedDouyin / TikTok platform tool for downloading works and collecting data
- douyin
- python
- tiktok
- api
TranslatedAutomatically upload videos to social media: Douyin, Xiaohongshu, Shipinhao (WeChat Channels), TikTok, YouTube, Bilibili
- bilibili
- douyin
- tiktok
- xiaohongshu
Proxy server to bypass Cloudflare protection
- puppeteer
- cloudflare
- proxy
- chrome
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
🚀 PR Agent: The Original Open-Source PR Reviewer. This project is not the Qodo free tier.
- openai
- pull-request
- code-review
- pull-requests
TranslatedDaidai Didi (带带弟弟, "take little brother along") general-purpose captcha recognition OCR, PyPI version
Deploy headless browsers in Docker. Run on our cloud or bring your own. Free for non-commercial uses.
C# application with primary purpose of farming Steam cards from multiple accounts simultaneously.
- steam
- steam-api
- steam-account
- steambot