Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
- webcrawler
- scrapy
- crawlab
- spiders-management
Repository Index
47,133 repositories indexed
1–24 of 26 · Most stars
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
Automate code & data workflows with interactive Elixir notebooks
- phoenix
- elixir
- notebooks
- collaborative
A simple, distributed task scheduler and runner with a web based UI.
An actor-based Framework with network transparency for creating event-driven architecture in Golang. Inspired by Erlang. Zero dependencies.
- erlang
- golang
- elixir
- supervisor
Restate is the platform for building resilient applications that tolerate all infrastructure faults w/o the need for a PhD.
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
Distributed task queue with full async support
- asyncio
- python
- python3
- task-executor
Pluggable foundation blocks for building distributed apps.
- azure
- c-sharp
- lock
- message-bus
🤖 Autonomous agent framework for Elixir. Built for distributed, autonomous behavior and dynamic workflows.
- elixir
- workflow
- event-driven-architecture
- orchestration
Fast and distributed workflow runner using ActiveJob and Redis
- workflows
- ruby
- sidekiq
- graph
"Security Incidents In A Box!" A modular, menu-driven, cross-platform tool for building customized, time-delayed, distributed security events. Easily create custom event chains for Blue- & Red Team drills and sensor / alert mapping. Red Teams can create decoy incidents, distractions, and lures to support and scale their operations. Build event sequences ("narratives") to simulate realistic scenarios and generate corresponding network and filesystem artifacts.
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
RedisSMQ: A high-performance Redis message queue for Node.js
- redis
- background-jobs
- brokers
- dead-letter-queue
iWF is a Workflow-As-Code microservice orchestration platform offering an orchestration coding framework and service for building resilient, fault-tolerant, scalable long-running processes
- golang
- workflow
- distributed-system
- orchestration
Easy C# API for Distributed Background Tasks/Jobs for .NET Core.
- netcore
- csharp
- aspnet
- netstandard
Maze Applied Reinforcement Learning Framework
The type-safe, event-driven workflow orchestration library that scales with your business. Build robust, distributed workflows in Go with compile-time safety, automatic retries, and horizontal scaling out of the box. Integrate with your existing tech stack using adapters 🚀
- eventdrivenarchitecture
- golang
- kafka
- reflex
state machine which runs code
- golang
- workflows
- state-machine
- actor-model
Ratus is a RESTful asynchronous task queue server. It translated concepts of distributed task queues into a set of resources that conform to REST principles and provides a consistent HTTP API for various backends.
- distributed-systems
- priority-queue
- task-queue
- task-scheduler
Distributed batch data processing framework
Durable execution framework for AI agents in Go. Keeps agent state, tool calls, and execution loops resilient across process crashes.
- agentic-ai
- ai
- ai-agents
- golang
Parallel Workflow extension for Rake, runs on multicores, clusters, clouds.
- ruby
- rake
- pwrake
- parallel
Distributed Automation for XNAT
- xnat
- python
- dax