Repository Index

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code

TranslatedAlibaba's MySQL binlog incremental subscription & consumption component

Apache Flink

• scala
• java
• big-data
• flink

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science

All-in-one IoT Platform - Device management, data collection, processing and visualization.

• visualization
• iot
• iot-platform
• mqtt

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql

GoDS (Go Data Structures) - Sets, Lists, Stacks, Maps, Trees, Queues, and much more

• go
• golang
• data-structure
• map

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

• metadata
• datadiscovery
• dataquality
• data-profiling

Open source RabbitMQ: core server and tier 1 (built-in) plugins

• rabbitmq
• amqp
• messaging
• amqp-0-9-1

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

• pandas-profiling
• pandas-dataframe
• statistics
• jupyter-notebook

Open-Source Web UI for Apache Kafka Management

• kafka-ui
• kafka-brokers
• kafka
• kafka-streams

CMAK is a tool for managing Apache Kafka clusters

• kafka
• scala
• cluster-management
• big-data

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

• websocket
• redis
• real-time
• messaging

Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

• kafka
• diskless
• automq
• serverless

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

• bloom
• deep-learning
• distributed-systems
• language-models

Building event-driven applications the easy way in Go.

• watermill
• event-driven
• event-sourcing
• sagas

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics

Fancy stream processing made operationally mundane

• message-queue
• stream-processing
• streaming-data
• message-bus

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

• steampipe
• postgresql
• postgresql-fdw
• cloud

Distributed Application Framework for .NET

• masstransit
• rabbitmq
• c-sharp
• azure-service-bus