Repository Index

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

Apache Kafka - A distributed event streaming platform

• scala
• kafka
• java
• streaming

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code

TranslatedAlibaba's MySQL binlog incremental subscription & consumption component

Apache Flink

• scala
• java
• big-data
• flink

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.

• rocketmq
• cloud-native
• eventing
• java

All-in-one IoT Platform - Device management, data collection, processing and visualization.

• visualization
• iot
• iot-platform
• mqtt

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql

TranslatedDataX is the open-source version of Alibaba Cloud DataWorks Data Integration.

Apache Hadoop

• hadoop

Apache Pulsar - distributed pub-sub messaging system

• pulsar
• pubsub
• messaging
• streaming

Logstash - transport and process your logs, events, or other data

• etl-framework
• streaming
• logging
• java

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler

Open source RabbitMQ: core server and tier 1 (built-in) plugins

• rabbitmq
• amqp
• messaging
• amqp-0-9-1

Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots

• chat
• chat-application
• chatapp
• go

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium

Open-Source Web UI for Apache Kafka Management

• kafka-ui
• kafka-brokers
• kafka
• kafka-streams

Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

• kafka
• diskless
• automq
• serverless

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time

Apache Iceberg

• iceberg
• apache
• hacktoberfest

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

• messaging
• java
• c-plus-plus
• ipc

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam

Pentaho Data Integration ( ETL ) a.k.a Kettle