Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Repository Index
65,264 visible repositories in the index
1–24 of 120 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Apache Kafka - A distributed event streaming platform
Apache Flink
Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
- rocketmq
- cloud-native
- eventing
- java
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Apache Hadoop
Apache Pulsar - distributed pub-sub messaging system
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
- workflow-schedule
- azkaban
- airflow
- task-scheduler
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
Sarama is a Go library for Apache Kafka.
- kafka
- kafka-client
- go
Open-Source Web UI for Apache Kafka Management
- kafka-ui
- kafka-brokers
- kafka
- kafka-streams
CMAK is a tool for managing Apache Kafka clusters
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
- weak-supervision
- data-cleaning
- data-quality
- noisy-labels
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Apache Iceberg
- iceberg
- apache
- hacktoberfest
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
- machine-learning
- computer-vision
- data-format
- deep-learning
Dockerfile for Apache Kafka
Apache Storm
- storm
- apache
- distributed
- streaming
Flink CDC is a streaming data integration tool
- change-data-capture
- cdc
- batch
- data-integration
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
Upserts, Deletes And Incremental Processing on Big Data.
- hudi
- apachehudi
- datalake
- bigdata