Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Repository Index
140,194 visible repositories in the index
1–24 of 338 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Your lightweight open-source personal agent.
📨 The ultimate agentic social media scheduling tool 🤖
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
Sim is the collaborative workspace to build, deploy, and monitor AI agents and workflows. Used by 100,000+ builders.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
A browser extension for automating your browser by connecting blocks
- vue
- chrome-extension
- automation
- browser-extension
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants
- nlp
- machine-learning
- machine-learning-library
- bot
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
A next-generation crawling and spidering framework.
- crawler
- web-spider
- gocrawler
- spider-framework
A lightweight, extendable front-end developer tool for mobile web page.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
Build voice agents with open-source models
- ai
- assistant
- language-model
- machine-learning
TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool
Moshi is a speech-text foundation model and full-duplex spoken dialogue framework. It uses Mimi, a state-of-the-art streaming neural audio codec.
Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.
- perimeterx
- nopecha
- ai-agents
- anti-captcha
💬 Typebot is a powerful chatbot builder that you can self-host.
- form-builder
- chatbot
- conversational-bots
- chat-application
💼【AI找工作助手】全平台自动投简历脚本:(boss、前程无忧、猎聘、智联招聘)
- ai
- getjob
- job-application
- job-search
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
- apify
- automation
- beautifulsoup
- crawler
Captcha solver extension for humans, available for Chrome, Edge and Firefox
- browser-extension
- chrome-extension
- firefox-extension
- captcha