Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
- webcrawler
- scrapy
- crawlab
- spiders-management
Repository Index
47,133 repositories indexed
1–24 of 33 · Most stars
A distributed transaction framework, supports workflow, saga, tcc, xa, 2-phase message, outbox patterns, supports many languages.
- saga
- xa
- tcc
- microservice
The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.
- cloud-computing
- deep-learning
- gpu
- hyperparameter-tuning
Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
- uber
- cadence
- workflows
- orchestration-engine
The complete load testing platform. Everything you need for production-grade load tests. Serverless & distributed. Load test with Playwright. Load test HTTP APIs, GraphQL, WebSocket, and more. Use any Node.js module.
Provides a simple way to run Selenium Grid with Chrome, Firefox, and Edge using Container Platform, making it easier to perform browser automation at scale
🪓 An orchestration engine for background tasks, AI agents, and durable workflows
Machinery is an asynchronous task queue/job queue based on distributed message passing.
- go
- golang
- task
- task-scheduler
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
Automate code & data workflows with interactive Elixir notebooks
- phoenix
- elixir
- notebooks
- collaborative
A simple, distributed task scheduler and runner with a web based UI.
An actor-based Framework with network transparency for creating event-driven architecture in Golang. Inspired by Erlang. Zero dependencies.
- erlang
- golang
- elixir
- supervisor
Restate is the platform for building resilient applications that tolerate all infrastructure faults w/o the need for a PhD.
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
Distributed task queue with full async support
- asyncio
- python
- python3
- task-executor
Pluggable foundation blocks for building distributed apps.
- azure
- c-sharp
- lock
- message-bus
🤖 Autonomous agent framework for Elixir. Built for distributed, autonomous behavior and dynamic workflows.
- elixir
- workflow
- event-driven-architecture
- orchestration
Fast and distributed workflow runner using ActiveJob and Redis
- workflows
- ruby
- sidekiq
- graph
"Security Incidents In A Box!" A modular, menu-driven, cross-platform tool for building customized, time-delayed, distributed security events. Easily create custom event chains for Blue- & Red Team drills and sensor / alert mapping. Red Teams can create decoy incidents, distractions, and lures to support and scale their operations. Build event sequences ("narratives") to simulate realistic scenarios and generate corresponding network and filesystem artifacts.
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
RedisSMQ: A high-performance Redis message queue for Node.js
- redis
- background-jobs
- brokers
- dead-letter-queue
iWF is a Workflow-As-Code microservice orchestration platform offering an orchestration coding framework and service for building resilient, fault-tolerant, scalable long-running processes
- golang
- workflow
- distributed-system
- orchestration
Easy C# API for Distributed Background Tasks/Jobs for .NET Core.
- netcore
- csharp
- aspnet
- netstandard
Maze Applied Reinforcement Learning Framework