Scrapy, a fast high-level web crawling & scraping framework for Python.
- python
- scraping
- crawling
- framework
Repository Index
47,133 repositories indexed
1–24 of 28 · Most stars
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
- browser
- rust
- servo
- browser-automation
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.
- golang
- chrome
- chrome-debugging-protocol
- testing
Incredibly fast crawler designed for OSINT.
TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool
Connect APIs, remarkably fast. Free for developers.
- eventsourcing
- event-sourcing
- automation
- workflow
Blazing fast deployer and server automation tool
- ruby
- rails
- mina
- deployment
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
Fast website link checker in Go
A framework to build Slack apps using Python
- slack
- slack-api
- bolt
- web-api
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
- meta-search-engine
- rust
- search-engine
- fast
Fast and distributed workflow runner using ActiveJob and Redis
- workflows
- ruby
- sidekiq
- graph
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Lightweight, fast and dependency-free Cron expression parser (due checker, next/prev due date finder), task runner, job scheduler and/or daemon for Golang (tested on v1.13+) and standalone usage. If you are bold, use it to replace crontab entirely.
- golang
- go
- crontab
- cronjob
GraphCommerce® is a headless storefront replacement for Magento 2 (PWA), that delivers a faster, better user experience. Fully customizable (React, Next.js) and open-source.
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
- golang
- web-crawler
- crawling
- scraping
A background task runner for crystal applications supporting periodic (CRON) and manually queued jobs
- crystal
- background-jobs
- fast
- redis
🕸️ Blazing fast GraphQL endpoints finder using subdomain enumeration, scripts analysis and bruteforce. 🕸️
Python/FastAPI application for monitoring Brazilian public-sector IT opportunities, with automated data collection, job filters, and official-source tracking.
Build beautiful Slack UIs fast
A blazing fast job processing system backed by GenStage & Redis.
- elixir-lang
- genstage
- background-jobs
- concurrent
Lightweight and lightning fast Java WebDriver client for test automation professionals