71,372 visible repositories in the index
1–12 of 12 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
zenoh unifies data in motion, data in-use, data at rest and computations. It carefully blends traditional pub/sub with geo-distributed storages, queries and computations, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks.
In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.
A high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary.
The fully compliant, embeddable high-performance Go MQTT v5 server for IoT, smarthome, and pubsub
Time-series database and pub/sub messaging server for IoT and real-time devices, in Go: an embeddable storage engine with TTLs, wildcard topics and encryption, and a clustered server over gRPC, TCP and WebSocket with replication and rolling upgrades.
Open-source streaming SQL engine written in Rust using Apache Arrow and DataFusion. Supports continuous queries, temporal stream joins, tumbling/session windows, and CDC/Kafka connectors. Lightweight, embeddable, and sub-microsecond latency
Stream Ops is a fully embeddable data streaming engine and stream processing API for Java.