1.OpenDCAI / DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
- data-science
- data-synthesis
- +7 more
+120 stars
from 8,313 · score 4.80
Trending repositories
Ranked by Attention Score: stars gained over the last 7 complete days
1–100 of 105 by Attention Score · Oct 28, 2026 – Nov 3, 2026 (UTC)
2.google-ai-edge / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
- mediapipe
- c-plus-plus
- computer-vision
- deep-learning
- android
- video-processing
- +11 more
+108 stars
from 37,431 · score 4.69
3.kestra-io / kestra
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
- automation
- devops
- +14 more
+103 stars
from 30,039 · score 4.64
4.open-metadata / OpenMetadata
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
- metadata
- datadiscovery
- dataquality
- data-profiling
- metadata-management
- data-catalog
- +14 more
+71 stars
from 15,601 · score 4.28
5.apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
+65 stars
from 47,290 · score 4.19
6.ray-project / ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
+63 stars
from 44,132 · score 4.16
7.apache / spark
Apache Spark - A unified analytics engine for large-scale data processing
+60 stars
from 44,346 · score 4.11
8.nats-io / nats-server
High-Performance server for NATS.io, the cloud and edge native messaging system.
- go
- golang
- messaging
- message-bus
- message-queue
- cloud-native
- +9 more
+54 stars
from 20,940 · score 4.01
9.cocoindex-io / cocoindex
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
- indexing
- python
- +14 more
+52 stars
from 11,766 · score 3.97
10.Unstructured-IO / unstructured
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
- ocr
- information-retrieval
- +14 more
+48 stars
from 15,629 · score 3.89
11.debezium / debezium
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
- cdc
- database
- +5 more
+34 stars
from 13,278 · score 3.56
12.dlt-hub / dlt
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
- data
- python
- data-engineering
- data-lake
- data-loading
- data-warehouse
- +4 more
+31 stars
from 6,020 · score 3.47
13.dbt-labs / dbt
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
- business-intelligence
- analytics
- +1 more
+26 stars
from 14,090 · score 3.30
+25 stars
from 9,379 · score 3.26
15.dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
- workflow-automation
- python
- +9 more
+24 stars
from 16,341 · score 3.22
16.openimsdk / open-im-server
IM Chat OpenClaw
+21 stars
from 16,731 · score 3.09
17.PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
- python
- workflow
- data-engineering
- data-science
- workflow-engine
- prefect
- +8 more
+20 stars
from 24,087 · score 3.04
18.centrifugal / centrifugo
Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.
+20 stars
from 10,926 · score 3.04
19.apache / seatunnel
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
- apache
- batch
- +8 more
+20 stars
from 9,765 · score 3.04
20.apache / iggy
Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed
+19 stars
from 5,216 · score 3.00
21.Netflix / metaflow
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
- ml-platform
- ml-infrastructure
- +14 more
+18 stars
from 10,320 · score 2.94
22.evidentlyai / evidently
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
- model-monitoring
- html-report
- +8 more
+18 stars
from 8,016 · score 2.94
23.slothflowlabs / duckle
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
- duckdb
- elt
- +14 more
+16 stars
from 1,532 · score 2.83
+13 stars
from 6,278 · score 2.64
+12 stars
from 26,431 · score 2.56
26.apache / dolphinscheduler
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
- workflow-schedule
- azkaban
- airflow
- task-scheduler
- job-scheduler
- cloud-native
- +5 more
+12 stars
from 14,538 · score 2.56
27.aeron-io / aeron
Efficient reliable UDP unicast, UDP multicast, and IPC message transport
- messaging
- java
- c-plus-plus
- ipc
- multicast-streams
- c
+12 stars
from 8,936 · score 2.56
28.apache / gravitino
World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
- datalake
- lakehouse
- metadata
- federated-query
- stratosphere
- metalake
- +5 more
+11 stars
from 3,335 · score 2.48
+9 stars
from 15,700 · score 2.30
30.apache / datafusion-ballista
Apache DataFusion Ballista Distributed Query Engine
- arrow
- big-data
- dataframe
- olap
- python
- query-engine
- +3 more
+9 stars
from 2,157 · score 2.30
31.bruin-data / bruin
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
- python
- snowflake
- +5 more
+9 stars
from 1,790 · score 2.30
32.fivetran / great_expectations
Always know what to expect from your data.
- pipeline-tests
- dataquality
- datacleaning
- datacleaner
- data-science
- data-profiling
- +14 more
+8 stars
from 11,906 · score 2.20
33.cleanlab / cleanlab
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
- weak-supervision
- data-cleaning
- data-quality
- noisy-labels
- data-centric-ai
- out-of-distribution-detection
- +14 more
+8 stars
from 11,721 · score 2.20
34.delta-io / delta
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
+8 stars
from 9,058 · score 2.20
35.apache / beam
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
- batch
- golang
- +2 more
+8 stars
from 8,697 · score 2.20
36.apache / camel
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
- cloud-native
- data-transformation
- +10 more
+8 stars
from 6,382 · score 2.20
37.ArroyoSystems / arroyo
Distributed stream processing engine in Rust
- data
- dev-tools
- infrastructure
- kafka
- rust
- sql
- +3 more
+8 stars
from 5,069 · score 2.20
38.risingwavelabs / risingwave
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
- database
- stream-processing
- rust
- postgresql
- kafka
- materialized-view
- +5 more
+7 stars
from 9,390 · score 2.08
39.zenml-io / zenml
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
- mlops
- machine-learning
- data-science
- production-ready
- devops-tools
- zenml
- +14 more
+7 stars
from 5,628 · score 2.08
40.ag2ai / faststream
Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.
- asyncio
- kafka
- python
- rabbitmq
- asyncapi
- distributed-systems
- +7 more
+6 stars
from 5,385 · score 1.95
41.bytewax / bytewax
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
- data-processing
- data-science
- +3 more
+6 stars
from 2,064 · score 1.95
42.alibaba / canal
TranslatedAlibaba's MySQL binlog incremental subscription & consumption component
+5 stars
from 29,739 · score 1.79
43.apache / kyuubi
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
+5 stars
from 2,371 · score 1.79
44.timeplus-io / proton
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
- single-binary
- stream-processing
- +10 more
+5 stars
from 2,276 · score 1.79
45.apache / rocketmq
Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
- rocketmq
- cloud-native
- eventing
- java
- messaging
- streaming
- +1 more
+4 stars
from 22,645 · score 1.61
46.feast-dev / feast
The Open Source Feature Store for AI/ML
- machine-learning
- features
- ml
- big-data
- feature-store
- python
- +4 more
+4 stars
from 7,349 · score 1.61
47.nats-io / nats.go
Golang client for NATS, the cloud native messaging system.
- go
- golang
- nats
- microservices
- microservices-architecture
- pub-sub
- +3 more
+4 stars
from 6,793 · score 1.61
48.apache / hudi
Upserts, Deletes And Incremental Processing on Big Data.
- hudi
- apachehudi
- datalake
- bigdata
- apachespark
- incremental-processing
- +3 more
+4 stars
from 6,344 · score 1.61
49.asyncapi / spec
The AsyncAPI specification allows you to create machine-readable definitions of your asynchronous APIs.
- asyncapi
- reactive
- specification
- websockets
- amqp
- mqtt
- +2 more
+4 stars
from 5,330 · score 1.61
50.apache / hamilton
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
- dataframe
- etl
- +13 more
+4 stars
from 2,618 · score 1.61
51.eclipse-iceoryx / iceoryx
Eclipse iceoryx™ - true zero-copy inter-process-communication
- iceoryx
- eclipse
- ipc
- inter-process-communication
- cpp
- zero-copy
- +10 more
+4 stars
from 2,202 · score 1.61
52.thingsboard / thingsboard-gateway
Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.
+4 stars
from 2,201 · score 1.61
53.lf-edge / ekuiper
Lightweight data stream processing engine for IoT edge
+4 stars
from 1,743 · score 1.61
54.OpenHFT / Chronicle-Queue
Micro second messaging that stores everything to disk
- chronicle
- queue
- java
- performance
- low-latency
- persistance
+3 stars
from 3,824 · score 1.39
55.Wisser / Jailer
Database Subsetting and Relational Data Browsing Tool.
+3 stars
from 3,213 · score 1.39
56.softwaremill / elasticmq
In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.
- scala
- sqs-interface
- elasticmq
- amazon-sqs
- aws
- aws-sqs
- +1 more
+3 stars
from 2,950 · score 1.39
57.numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
- pipeline
- map-reduce
- +1 more
+3 stars
from 2,834 · score 1.39
58.GoogleCloudPlatform / professional-services-data-validator
Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
+3 stars
from 532 · score 1.39
59.pingcap / tiflow
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
+3 stars
from 462 · score 1.39
60.MassTransit / MassTransit
Distributed Application Framework for .NET
+2 stars
from 7,811 · score 1.10
61.apache / ignite
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
- ignite
- data-management-platform
- +9 more
+2 stars
from 5,088 · score 1.10
62.meetfranz / franz
Franz is a free messaging app for services like WhatsApp, Slack, Messenger and many more.
+2 stars
from 4,727 · score 1.10
63.RoaringBitmap / RoaringBitmap
A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others
- java
- bitset
- roaring-bitmaps
- roaringbitmap
- druid
- spark
- +1 more
+2 stars
from 3,954 · score 1.10
64.vernemq / vernemq
A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases. The VerneMQ mission is active & the project maintained. Thank you for your support!
- mqtt
- m2m
- scalable
- broker
- erlang
- distributed
- +13 more
+2 stars
from 3,644 · score 1.10
65.WeBankFinTech / DataSphereStudio
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
+2 stars
from 3,273 · score 1.10
66.bloomberg / blazingmq
A modern high-performance open source message queuing system
+2 stars
from 3,219 · score 1.10
67.absmach / magistrala
IoT Platform Framework
- golang
- iot-platform
- edge-computing
- distributed-systems
- iot-cloud
- iot-gateway
- +14 more
+2 stars
from 2,656 · score 1.10
68.quixio / quix-streams
Python Streaming DataFrames for Kafka
+2 stars
from 1,580 · score 1.10
69.projectnessie / nessie
Nessie: Transactional Catalog for Data Lakes with Git-like semantics
- data
- spark
- java
- iceberg
- aws-lambda
- git
+2 stars
from 1,524 · score 1.10
70.PatMartin / Dex
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.
- data-science
- data-visualization
- visualization
- data-analysis
- data-mining
- java
- +7 more
+2 stars
from 1,316 · score 1.10
71.spring-cloud / spring-cloud-stream
Framework for building Event-Driven Microservices
- microservices
- messaging
- event-driven-microservices
- stream-processing
- reactive
- cloud-native
+2 stars
from 1,070 · score 1.10
72.thingsboard / tbmq
The ultimate distributed MQTT broker. Handles 100M+ connections and 10M msg/sec with ease. Built on Kafka to provide industrial-grade persistence and eliminate data loss.
+2 stars
from 774 · score 1.10
73.apache / streampipes
Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.
- iot
- analytics
- edge
- stream-processing
- iiot
- self-service
- +6 more
+2 stars
from 754 · score 1.10
74.samber / ro
🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications
- event-driven
- generics
- go
- golang
- pipe
- pipeline
- +11 more
+2 stars
from 694 · score 1.10
75.provectus / kafka-ui
Open-Source Web UI for Apache Kafka Management
- kafka-ui
- kafka-brokers
- kafka
- kafka-streams
- kafka-client
- opensource
- +12 more
+1 stars
from 12,313 · score 0.69
76.snowplow / snowplow
The leader in Customer Data Infrastructure
- snowplow
- analytics
- data
- data-pipeline
- data-collection
- product-analytics
- +3 more
+1 stars
from 7,047 · score 0.69
+1 stars
from 6,698 · score 0.69
78.polyaxon / polyaxon
AI Infra / AI Orchestration / AI Control Plane
+1 stars
from 3,746 · score 0.69
79.awslabs / deequ
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
- dataquality
- spark
- unit-testing
- scala
+1 stars
from 3,656 · score 0.69
80.apache / linkis
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
+1 stars
from 3,415 · score 0.69
81.alibaba / pipcook
Machine learning platform for Web developers
+1 stars
from 2,593 · score 0.69
+1 stars
from 2,462 · score 0.69
83.apache / sedona
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
- spatial-query
- scala
- +3 more
+1 stars
from 2,425 · score 0.69
84.elastic / elasticsearch-hadoop
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
+1 stars
from 1,966 · score 0.69
85.Netflix / genie
Distributed Big Data Orchestration Service
- big-data
- bigdata
- orchestration
- configuration
- configuration-management
- java
- +7 more
+1 stars
from 1,767 · score 0.69
86.apache / eventmesh
EventMesh is a new generation serverless event middleware for building distributed event-driven applications.
- esb
- event-driven
- event-mesh
- event-governance
- cloud-native
- serverless
- +11 more
+1 stars
from 1,758 · score 0.69
87.WallarooLabs / wally
Distributed Stream Processing
- stream-processing
- stream-processor
- stream-processing-engine
- pony-language
- python
- linux
- +4 more
+1 stars
from 1,485 · score 0.69
+1 stars
from 1,405 · score 0.69
89.arkflow-rs / arkflow
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
- sql
- datafusion
- +14 more
+1 stars
from 1,315 · score 0.69
90.graphframes / graphframes
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
- apache-spark
- big-data
- connected-components
- dataframe
- dataframes
- graphs
- +8 more
+1 stars
from 1,211 · score 0.69
91.nanomsg / mangos
mangos is a pure Golang implementation of nanomsg's "Scalablilty Protocols"
+1 stars
from 761 · score 0.69
92.mrpowers-io / quinn
pyspark methods to enhance developer productivity 📣 👯 🎉
+1 stars
from 689 · score 0.69
+1 stars
from 682 · score 0.69
94.capitalone / datacompy
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
- python
- pandas
- spark
- data
- data-science
- compare
- +8 more
+1 stars
from 664 · score 0.69
95.zarusz / SlimMessageBus
Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.
+1 stars
from 660 · score 0.69
+1 stars
from 447 · score 0.69
97.wso2 / product-ei
An open source, a high-performance hybrid integration platform that allows developers quick integration with any application, data, or system.
- integration
- middleware
- wso2
- esb
- messaging
- business-process
- +5 more
+1 stars
from 402 · score 0.69
98.reductstore / reductstore
High-performance, time-indexed object storage for robotics and industrial IoT
- blob-storage
- storage-engine
- storage
- historian
- timeseries
- time-series
- +9 more
+1 stars
from 376 · score 0.69
99.Ahoo-Wang / Wow
TranslatedModern Reactive CQRS Architecture Microservice development framework based on DDD and EventSourcing
- cqrs
- domain-driven-design
- event-driven
- event-sourcing
- microservice
- reactive-programming
- +13 more
+1 stars
from 304 · score 0.69
100.Breeze0806 / go-etl
go-etl is a toolset for data extraction, transformation and loading.
- etl
- mysql
- go
- postgresql
- greenplum
- csv
- +5 more
+1 stars
from 191 · score 0.69