57,138 visible repositories in the index
1–24 of 29 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Scrapy, a fast high-level web crawling & scraping framework for Python.
Your lightweight open-source personal agent.
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
TranslatedA multi-task real-time/scheduled monitoring and intelligent analysis system for Xianyu (闲鱼), built with Playwright and AI, and equipped with a full-featured back-end management UI. It helps users find the products they like among the vast number of items on Xianyu.
Translated🤖 DIY-able multimodal AI chatbot | 🚀 Quickly connects to chat platforms such as WeChat, QQ, Telegram, and more | 🦈 Supports DeepSeek, Grok, Claude, Ollama, Gemini, OpenAI | Workflow system, web search, AI image generation, persona tuning, virtual maid, voice conversation |
Python version of the Playwright testing and automation library.
Python APIs for web automation, testing, and bypassing bot-detection.
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
Lighter web automation with Python
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
The first AI agent that builds permissionless integrations through reverse engineering platforms' internal APIs.
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
A chat bot for Slack (https://slack.com).
Free and open-source employee onboarding platform. Onboard new hires through Slack or the web portal.
:scissors: High performance, multi-threaded image scraper
Translated📚 WeChat Reading (微信读书) lossless full-book export tool — one-click export of dual-font, finely typeset PDF / EPUB / plain Markdown notes (compatible with Obsidian & Notion), solving the problem of missing characters when turning pages on the web version.
Translated✈️ Self-hosted multi-channel airfare price monitoring: flight detail collection → DingTalk rich-media push → four-layer strong alerts when the target price is met. Domestic routes and international routes (destinations on five continents) share the same collect-judge-push pipeline; data stays on your own machine, all configuration is done on a web page, open source and free.
A lightweight system to automate posting to all of your socials from one dashboard
Automating captcha solving for Playwright. Supports Cloudflare Turnstile & Interstitial, and reCAPTCHA V2 & V3 with Click-based or API (2captcha) solving. Designed for easy integration with Playwright, Patchright, and Camoufox.
A simple yet highly configurable Mastodon bot that posts geotagged aerial imagery of a random location in the world.