47,133 repositories indexed
1–24 of 45 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Scrapy, a fast high-level web crawling & scraping framework for Python.
📨 The ultimate agentic social media scheduling tool 🤖
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Elegant Scraper and Crawler Framework for Golang
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/products/free-netbox-cloud/
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Turn any website into a structured API. Extract, automate, search and monitor the web.
A next-generation crawling and spidering framework.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
TranslatedWeChat assistant: 1. Sends customized messages to friends (a girlfriend) at scheduled times every day. 2. A bot automatically replies to friends. 3. Group assistant features (for example: looking up garbage sorting, weather, calendar, real-time movie box office, express delivery tracking, PM2.5, etc.)
✨ The all-contributors bot website and documentation. Recognize all contributors, not just the ones who push code ✨
A Chrome DevTools Protocol driver for web automation and scraping.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
CRON for PHP: Calculate the next or previous run date and determine if a CRON expression is due
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Headless Chrome .NET API
A browser testing and web crawling library for PHP and Symfony
Native UI testing / controlling with node
CLI tool to automate docker image updates. Interactive or unattended with notifications, image backups, autoprune, no pre-pulling and more.
Async Python 3.6+ web scraping micro-framework based on asyncio