Repository Index

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning

Cross-platform, customizable ML solutions for live and streaming media.

• mediapipe
• c-plus-plus
• computer-vision
• deep-learning

High-Performance server for NATS.io, the cloud and edge native messaging system.

• go
• golang
• messaging
• message-bus

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

• python
• luigi
• orchestration-framework
• scheduling

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp

Open source RabbitMQ: core server and tier 1 (built-in) plugins

• rabbitmq
• amqp
• messaging
• amqp-0-9-1

Parallel computing with task scheduling

• dask
• python
• pydata
• numpy

Node-based Visual Programming Toolbox

• photogrammetry
• computer-vision
• 3d-reconstruction
• alicevision

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

• kafka
• cpp
• kubernetes
• go

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

• bloom
• deep-learning
• distributed-systems
• language-models

Build, Manage and Deploy AI/ML Systems

• machine-learning
• model-management
• ai
• ml

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

• java
• hazelcast
• in-memory
• big-data

The AsyncAPI specification allows you to create machine-readable definitions of your asynchronous APIs.

• asyncapi
• reactive
• specification
• websockets

Apache Ignite

• distributed-sql-database
• iot
• osgi
• network-client

A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others

• java
• bitset
• roaring-bitmaps
• roaringbitmap

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

• distributed-computing
• map-reduce
• golang
• distributed-systems

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

• sql
• spark
• hive
• pyspark

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.

• scala
• sqs-interface
• elasticmq
• amazon-sqs

IoT Platform Framework

• golang
• iot-platform
• edge-computing
• distributed-systems

An ultra-lightweight and blazing-fast Messaging Bus/MQTT Broker for Edge & SDV

• mqtt
• mqtt-broker
• iot-edge
• nng

Apache ActiveMQ

• activemq
• java
• amqp
• amqps