86,076 visible repositories in the index
1–24 of 93 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Scrapy, a fast high-level web crawling & scraping framework for Python.
🧡 Everything is RSSible
🕵️♂️ Collect a dossier on a person by username from 6K websites
Elegant Scraper and Crawler Framework for Golang
A next-generation crawling and spidering framework.
TranslatedDouyin / TikTok platform tool for downloading works and collecting data
Download pictures (or videos) along with their captions and other metadata from Instagram.
TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
A scalable web crawler framework for Java.
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
TranslatedGet the real streaming URLs (live stream sources) and danmaku (live comments) of 58 live streaming platforms, including Douyu, Huya, Bilibili, Douyin, and Kuaishou. The live stream sources can be played in players such as PotPlayer and flv.js.
Turn any webpage into structured data using LLMs
The All in One Framework to Build Undefeatable Scrapers
Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha
Downloads albums from the web in bulk for archive purposes
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
新闻网页正文通用抽取器 Beta 版.
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
Shadow any website for offline viewing, with the JavaScript stripped out