Repository Index

Apache Superset is a Data Visualization and Data Exploration Platform

• superset
• apache
• apache-superset
• data-visualization

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python

This is a repo with links to everything you'd ever want to learn about data engineering

• apachespark
• awesome
• bigdata
• data

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

Apache Kafka - A distributed event streaming platform

• scala
• kafka
• java
• streaming

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

• machine-learning
• ai
• ml
• mlflow

Apache Flink

• scala
• java
• big-data
• flink

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.

• rocketmq
• cloud-native
• eventing
• java

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql

QuestDB is a high performance, open-source, time-series database

• time-series
• low-latency
• database
• sql

Apache Doris is a real-time analytics and hybrid search database for AI agents.

• olap
• database
• hudi
• iceberg

Apache Hadoop

• hadoop

Apache Pulsar - distributed pub-sub messaging system

• pulsar
• pubsub
• messaging
• streaming

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium

Sarama is a Go library for Apache Kafka.

• kafka
• kafka-client
• go

Open-Source Web UI for Apache Kafka Management

• kafka-ui
• kafka-brokers
• kafka
• kafka-streams

CMAK is a tool for managing Apache Kafka clusters

• kafka
• scala
• cluster-management
• big-data

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

• dataframe
• python
• bigdata
• tabular-data

Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability

• content
• network-server
• http
• cloud

Dockerfile for Apache Kafka