Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Repository Index
47,133 repositories indexed
1–24 of 113 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Apache Kafka - A distributed event streaming platform
Apache Flink
Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
- rocketmq
- cloud-native
- eventing
- java
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Apache Hadoop
Apache Pulsar - distributed pub-sub messaging system
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
- workflow-schedule
- azkaban
- airflow
- task-scheduler
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
Sarama is a Go library for Apache Kafka.
- kafka
- kafka-client
- go
Open-Source Web UI for Apache Kafka Management
- kafka-ui
- kafka-brokers
- kafka
- kafka-streams
CMAK is a tool for managing Apache Kafka clusters
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
- dataframe
- python
- bigdata
- tabular-data
Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability
Dockerfile for Apache Kafka
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
Upserts, Deletes And Incremental Processing on Big Data.
- hudi
- apachehudi
- datalake
- bigdata
Generic command line non-JVM Apache Kafka producer and consumer
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark