47,133 repositories indexed
25–48 of 683 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
Apache Pulsar - distributed pub-sub messaging system
Logstash - transport and process your logs, events, or other data
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
Open source RabbitMQ: core server and tier 1 (built-in) plugins
Parallel computing with task scheduling
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
Roadmap to becoming a data engineer in 2021
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
Sarama is a Go library for Apache Kafka.
A distributed, fast open-source graph database featuring horizontal scalability and high availability
A curated list of Domain-Driven Design (DDD), Command Query Responsibility Segregation (CQRS), Event Sourcing, and Event Storming resources
Open-Source Web UI for Apache Kafka Management
The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
CMAK is a tool for managing Apache Kafka clusters
Always know what to expect from your data.
🥊 Tiny 200 byte functional event emitter / pubsub.
Incremental engine for long horizon agents 🌟 Star if you like it!
Connect APIs, remarkably fast. Free for developers.