SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.
Repository Index
132,241 visible repositories in the index
1–24 of 134 · Most stars
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
- rocketmq
- cloud-native
- eventing
- java
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
- pandas-profiling
- pandas-dataframe
- statistics
- jupyter-notebook
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
Open-Source Web UI for Apache Kafka Management
- kafka-ui
- kafka-brokers
- kafka
- kafka-streams
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Apache Iceberg
- iceberg
- apache
- hacktoberfest
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also other languages (rust, go and ts)
- machine-learning
- scale
- workflow
- mlops
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
Apache KIE (Knowledge Is Everything) is the home of Drools, OptaPlanner, jBPM, and Kogito.
- rule-engine
- java
- dmn
- hacktoberfest
Apache NiFi
- nifi
- java
- hacktoberfest
- apache
Data-Centric Pipelines and Data Versioning
Kafka Web UI
- kafka
- kubernetes
- docker
- consumer-group
A Cloud Native Batch System (Project under CNCF)
- batch-systems
- kubernetes
- golang
- hpc
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Translated🚗 Technical talks I have given personally... (topics: high-performance architecture, middleware principles and practice, cloud native, Golang, etc.)
- ppt
- python
- golang
- keynote
Policy and data administration, distribution, and real-time updates on top of Policy Agents (OPA, Cedar, ...)
- authorization
- policy-as-code
- policy
- realtime
Reactive Extensions for the Go language.
- reactivex
- golang
- observable
- go