67,262 visible repositories in the index
1–24 of 37 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python scraper based on AI
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Download pictures (or videos) along with their captions and other metadata from Instagram.
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Translated:orange_book: Chinese Xinhua Dictionary database. Includes xiehouyu (two-part allegorical sayings), idioms, words, and Chinese characters.
TranslatedSina Weibo crawler, uses Python to scrape Sina Weibo data
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!
The AI local media metadata crawler
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Build databases, automations, apps & agents with AI — no code. Open source platform available on cloud and self-hosted. GDPR, HIPAA, SOC 2 compliant. Best Airtable alternative.
Agent-ready RPA suite with out-of-the-box automation tools. Built for individuals and enterprises.
Open-Source Unified Vulnerability Management, DevSecOps & ASPM
:zap: A distributed crawler for weibo, building with celery and requests.
TranslatedSina Weibo crawler: uses Python to crawl Sina Weibo data and download Weibo images and Weibo videos
A library that scrapes Linkedin for user data
Interact with your SQL database, Natural Language to SQL using LLMs
Chatbot继续沿着LLM前进，近期更新小参数量SLM的和训练脚本，支持本地训练。新增ChatAgent,实现各种有实际场景价值的Agent实现。
Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
OpenTalking: An industrial-grade open-source AI digital human framework that supports real-time conversation, private deployment, and pluggable models.
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.