47,133 repositories indexed
1–13 of 13 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Scrapy, a fast high-level web crawling & scraping framework for Python.
The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/products/free-netbox-cloud/
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
TranslatedWeChat assistant: 1. Sends customized messages to friends (a girlfriend) at scheduled times every day. 2. A bot automatically replies to friends. 3. Group assistant features (for example: looking up garbage sorting, weather, calendar, real-time movie box office, express delivery tracking, PM2.5, etc.)
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Async Python 3.6+ web scraping micro-framework based on asyncio
WarcDB: Web crawl data as SQLite databases.
SimpleCRON is a time-based task scheduling program inspired by the well-known CRON program for Unix systems.
Antsichaut aims to automate the filling of a changelog.yaml used by antsibull-changelog
Python toolkit for discovering, monitoring, and controlling Tasmota devices via MQTT with CLI utilities and ready-to-use multi-arch Docker images