47,133 repositories indexed
25–48 of 675 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
Apache Pulsar - distributed pub-sub messaging system
Logstash - transport and process your logs, events, or other data
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
Open source RabbitMQ: core server and tier 1 (built-in) plugins
Parallel computing with task scheduling
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
Node-based Visual Programming Toolbox
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
Sarama is a Go library for Apache Kafka.
Open-Source Web UI for Apache Kafka Management
CMAK is a tool for managing Apache Kafka clusters
Always know what to expect from your data.
🥊 Tiny 200 byte functional event emitter / pubsub.
Incremental engine for long horizon agents 🌟 Star if you like it!
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
Eclipse Mosquitto - An open source MQTT broker
ZeroMQ core engine in C++, implements ZMTP/3.1
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.