Scrapy, a fast high-level web crawling & scraping framework for Python.
- python
- scraping
- crawling
- framework
Repository Index
55,140 visible repositories in the index
1–15 of 15 · Most stars
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
- browser
- rust
- servo
- browser-automation
High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
- browser-automation
- cdp
- golang
- headless-chrome
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
Venom is a high-performance system developed with JavaScript to create a bot for WhatsApp, support for creating any interaction, such as customer service, media sending, sentence recognition based on artificial intelligence and all types of design architecture for WhatsApp.
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
A high-performance Internet Relay Chat (IRCv3) server for UNIX-like and Windows systems
- inspircd
- c-plus-plus
- irc-server
- ircv3
Crawly, a high-level web crawling & scraping framework for Elixir.
A high performance web crawler / scraper in Elixir.
:scissors: High performance, multi-threaded image scraper
High-performance HTML5 parser for Ruby based on Lexbor, with support for both CSS selectors and XPath.
A simple yet highly configurable Mastodon bot that posts geotagged aerial imagery of a random location in the world.
A high-level API to control headless Chrome or Chromium over the DevTools Protocol. It is the Rust equivalent of Puppeteer, a Node library maintained by the Chrome DevTools team.
SpiderMan,a base-on Broadway fast high-level web crawling & scraping framework for Elixir.
Seamlessly fetch paginated data from any source. Simple and high performance API scraping included!
- api-scraper
- indexer
- pagination
- paginator