55,140 visible repositories in the index
1–24 of 25 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python scraper based on AI
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Download pictures (or videos) along with their captions and other metadata from Instagram.
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Translated:orange_book: Chinese Xinhua Dictionary database. Includes xiehouyu (two-part allegorical sayings), idioms, words, and Chinese characters.
Moshi is a speech-text foundation model and full-duplex spoken dialogue framework. It uses Mimi, a state-of-the-art streaming neural audio codec.
TranslatedSina Weibo crawler, uses Python to scrape Sina Weibo data
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!
The AI local media metadata crawler
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Build databases, automations, apps & agents with AI — no code. Open source platform available on cloud and self-hosted. GDPR, HIPAA, SOC 2 compliant. Best Airtable alternative.
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
WarcDB: Web crawl data as SQLite databases.
Send Sir Perceval on a quest to retrieve and gather data from software repositories.
Z-Hound — a CSV-first collector that finds public threat reports and extracts structured IoCs without a database, LLM, or API key.
Python/FastAPI application for monitoring Brazilian public-sector IT opportunities, with automated data collection, job filters, and official-source tracking.
Credential Phish Analysis and Automation
Scrapy project for data capture of vgchartz
Airut is a system for running Claude Code tasks from email and Slack. It handles workspace provisioning, container isolation, network sandboxing, session persistence, and cleanup — a secure foundation for autonomous agentic development.
The Raspberry Pi Telegram Bot is a lightweight, customizable bot built for Raspberry Pi users. It leverages Python and the Telegram Bot API to enable users to execute commands, receive updates, and automate processes from their Telegram account. Ideal for DIY projects, home automation, and remote control applications, this bot is easy to set up .