Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Repository Index
47,133 repositories indexed
1–13 of 13 · Most stars
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
A light-weight, flexible, and expressive statistical data testing library
- pandas
- validation
- schema
- dataframes
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Stream Processing and Complex Event Processing Engine
- streaming-data-processing
- cep
- stream-processing
- complex-event-processing
Gmqtt is a flexible, high-performance MQTT broker library that fully implements the MQTT protocol V3.x and V5 in golang
- mqtt
- mqtt-broker
- mqtt-protocol
- mqtt-server
Esper Complex Event Processing, Streaming SQL and Event Series Analysis
- cep
- complex-event-processing
- open-source
- compiler
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
- golang
- event-sourcing
- stream-processing
- brokers
Postgres change events (CDC) in Elixir [WAL]
- cdc
- change-data-capture
- elixir
- postgres
Highly optimized, extremely modular and very flexible XMPP/Jabber server
- tigase
- xmpp
- server
- java
Real-time decision features without streaming infra. Turn live events into product reflexes — no Kafka, no Flink, no feature store.
- analytics
- feature-store
- fraud-detection
- python
A library for scheduling when to dispatch a message to a channel