Trending repositories

1.OpenDCAI / DataFlow

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

• data
• data-cleaning
• data-pipelines
• data-processing
• data-science
• data-synthesis
• +7 more

2.google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

• mediapipe
• c-plus-plus
• computer-vision
• deep-learning
• android
• video-processing
• +11 more

3.kestra-io / kestra

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code
• automation
• devops
• +14 more

4.open-metadata / OpenMetadata

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

• metadata
• datadiscovery
• dataquality
• data-profiling
• metadata-management
• data-catalog
• +14 more

5.apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python
• scheduler
• workflow
• +14 more

6.ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning
• reinforcement-learning
• deep-learning
• +14 more

7.apache / spark

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java
• big-data
• jdbc
• +2 more

8.nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.

• go
• golang
• messaging
• message-bus
• message-queue
• cloud-native
• +9 more

9.cocoindex-io / cocoindex

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python
• +14 more

10.Unstructured-IO / unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp
• ocr
• information-retrieval
• +14 more

11.debezium / debezium

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium
• cdc
• database
• +5 more

12.dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake
• data-loading
• data-warehouse
• +4 more

13.dbt-labs / dbt

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

• dbt-viewpoint
• slack
• pypa
• data-modeling
• business-intelligence
• analytics
• +1 more

14.apache / iceberg

Apache Iceberg

• iceberg
• apache
• hacktoberfest

15.dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science
• workflow-automation
• python
• +9 more

16.openimsdk / open-im-server

IM Chat OpenClaw

• im
• chat
• messaging
• messenger
• openclaw

17.PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science
• workflow-engine
• prefect
• +8 more

18.centrifugal / centrifugo

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

• websocket
• redis
• real-time
• messaging
• scalability
• eventsource
• +14 more

19.apache / seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time
• apache
• batch
• +8 more

20.apache / iggy

Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed

• messaging
• quic
• rust
• streaming
• http
• iggy
• +2 more

21.Netflix / metaflow

Build, Manage and Deploy AI/ML Systems

• machine-learning
• model-management
• ai
• ml
• ml-platform
• ml-infrastructure
• +14 more

22.evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning
• model-monitoring
• html-report
• +8 more

23.slothflowlabs / duckle

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

• data-engineering
• data-integration
• data-pipeline
• data-quality
• duckdb
• elt
• +14 more

24.apache / nifi

Apache NiFi

• nifi
• java
• hacktoberfest
• apache

25.apache / flink

Apache Flink

• scala
• java
• big-data
• flink
• python
• sql

26.apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler
• job-scheduler
• cloud-native
• +5 more

27.aeron-io / aeron

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

• messaging
• java
• c-plus-plus
• ipc
• multicast-streams
• c

28.apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query
• stratosphere
• metalake
• +5 more

29.apache / hadoop

Apache Hadoop

• hadoop

30.apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap
• python
• query-engine
• +3 more

31.bruin-data / bruin

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines
• python
• snowflake
• +5 more

32.fivetran / great_expectations

Always know what to expect from your data.

• pipeline-tests
• dataquality
• datacleaning
• datacleaner
• data-science
• data-profiling
• +14 more

33.cleanlab / cleanlab

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

• weak-supervision
• data-cleaning
• data-quality
• noisy-labels
• data-centric-ai
• out-of-distribution-detection
• +14 more

34.delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics
• delta-lake

35.apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam
• batch
• golang
• +2 more

36.apache / camel

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices
• cloud-native
• data-transformation
• +10 more

37.ArroyoSystems / arroyo

Distributed stream processing engine in Rust

• data
• dev-tools
• infrastructure
• kafka
• rust
• sql
• +3 more

38.risingwavelabs / risingwave

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql
• kafka
• materialized-view
• +5 more

39.zenml-io / zenml

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

• mlops
• machine-learning
• data-science
• production-ready
• devops-tools
• zenml
• +14 more

40.ag2ai / faststream

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

• asyncio
• kafka
• python
• rabbitmq
• asyncapi
• distributed-systems
• +7 more

41.bytewax / bytewax

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering
• data-processing
• data-science
• +3 more

42.alibaba / canal

TranslatedAlibaba's MySQL binlog incremental subscription & consumption component

43.apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift
• jdbc
• spark-sql
• +4 more

44.timeplus-io / proton

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda
• single-binary
• stream-processing
• +10 more

45.microcks / microcks

The open source, cloud native tool for API Mocking and Testing. Microcks is a Cloud Native Computing Foundation incubating project 🚀

• api
• testing
• kubernetes
• api-testing
• event-driven
• graphql
• +14 more

46.apache / rocketmq

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.

• rocketmq
• cloud-native
• eventing
• java
• messaging
• streaming
• +1 more

47.feast-dev / feast

The Open Source Feature Store for AI/ML

• machine-learning
• features
• ml
• big-data
• feature-store
• python
• +4 more

48.nats-io / nats.go

Golang client for NATS, the cloud native messaging system.

• go
• golang
• nats
• microservices
• microservices-architecture
• pub-sub
• +3 more

49.apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.

• hudi
• apachehudi
• datalake
• bigdata
• apachespark
• incremental-processing
• +3 more

50.asyncapi / spec

The AsyncAPI specification allows you to create machine-readable definitions of your asynchronous APIs.

• asyncapi
• reactive
• specification
• websockets
• amqp
• mqtt
• +2 more

51.krakend / krakend-ce

KrakenD Community Edition: High-performance, stateless, declarative, API Gateway written in Go.

• api-gateway
• microservices
• backend-services
• reverse-proxy
• cloud-native
• microservice
• +8 more

52.apache / hamilton

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering
• dataframe
• etl
• +13 more

53.eclipse-iceoryx / iceoryx

Eclipse iceoryx™ - true zero-copy inter-process-communication

• iceoryx
• eclipse
• ipc
• inter-process-communication
• cpp
• zero-copy
• +10 more

54.lf-edge / ekuiper

Lightweight data stream processing engine for IoT edge

• edge
• sql
• iot
• edge-analytics
• edge-streaming
• edge-computing
• +2 more

55.OpenHFT / Chronicle-Queue

Micro second messaging that stores everything to disk

• chronicle
• queue
• java
• performance
• low-latency
• persistance

56.apache / hugegraph

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)

• database
• graph-database
• graph
• gremlin
• big-data
• graphdb

57.softwaremill / elasticmq

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.

• scala
• sqs-interface
• elasticmq
• amazon-sqs
• aws
• aws-sqs
• +1 more

58.numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

• kubernetes
• stream-processing
• data-processing
• k8s
• pipeline
• map-reduce
• +1 more

59.pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)

• cdc
• tidb
• mysql
• kafka
• ticdc
• dm

60.MassTransit / MassTransit

Distributed Application Framework for .NET

• masstransit
• rabbitmq
• c-sharp
• azure-service-bus
• messaging
• dotnet
• +2 more

61.apache / ignite

Apache Ignite

• distributed-sql-database
• iot
• osgi
• network-client
• ignite
• data-management-platform
• +9 more

62.meetfranz / franz

Franz is a free messaging app for services like WhatsApp, Slack, Messenger and many more.

• messaging
• electron
• franz
• messenger
• whatsapp
• slack
• +3 more

63.RoaringBitmap / RoaringBitmap

A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others

• java
• bitset
• roaring-bitmaps
• roaringbitmap
• druid
• spark
• +1 more

64.vernemq / vernemq

A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases. The VerneMQ mission is active & the project maintained. Thank you for your support!

• mqtt
• m2m
• scalable
• broker
• erlang
• distributed
• +13 more

65.apache / avro

Apache Avro is a data serialization system.

• csharp
• ruby
• cplusplus
• python
• php
• java
• +5 more

66.WeBankFinTech / DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

• workflow
• governance
• azkaban
• davinci
• linkis
• spark
• +14 more

67.bloomberg / blazingmq

A modern high-performance open source message queuing system

• distributed-systems
• message-queue
• messaging
• middleware
• queue
• queues
• +1 more

68.absmach / magistrala

IoT Platform Framework

• golang
• iot-platform
• edge-computing
• distributed-systems
• iot-cloud
• iot-gateway
• +14 more

69.elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.

• notebooks
• notebook-jupyter
• jupyterlab
• jupyterlab-extensions
• ai
• machine-learning
• +14 more

70.matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS

• aws
• cloud
• security
• big-data
• serverless
• apache-iceberg
• +14 more

71.quixio / quix-streams

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering
• data-science
• machine-learning
• +8 more

72.projectnessie / nessie

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

• data
• spark
• java
• iceberg
• aws-lambda
• git

73.apache / cloudberry

One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.

• database
• data-analysis
• data-warehouse
• olap
• postgresql
• sql
• +8 more

74.halaxa / json-machine

Efficient, easy-to-use, and fast PHP JSON stream parser

• php
• stream-processing
• json-parser
• json-iterator
• json-stream
• parsing

75.PatMartin / Dex

Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.

• data-science
• data-visualization
• visualization
• data-analysis
• data-mining
• java
• +7 more

76.spring-cloud / spring-cloud-stream

Framework for building Event-Driven Microservices

• microservices
• messaging
• event-driven-microservices
• stream-processing
• reactive
• cloud-native

77.thingsboard / tbmq

The ultimate distributed MQTT broker. Handles 100M+ connections and 10M msg/sec with ease. Built on Kafka to provide industrial-grade persistence and eliminate data loss.

• broker
• iot
• java
• kafka
• mqtt
• mqtt-broker
• +5 more

78.apache / streampipes

Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

• iot
• analytics
• edge
• stream-processing
• iiot
• self-service
• +6 more

79.samber / ro

🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications

• event-driven
• generics
• go
• golang
• pipe
• pipeline
• +11 more

80.provectus / kafka-ui

Open-Source Web UI for Apache Kafka Management

• kafka-ui
• kafka-brokers
• kafka
• kafka-streams
• kafka-client
• opensource
• +12 more

81.snowplow / snowplow

The leader in Customer Data Infrastructure

• snowplow
• analytics
• data
• data-pipeline
• data-collection
• product-analytics
• +3 more

82.apache / storm

Apache Storm

• storm
• apache
• distributed
• streaming

83.polyaxon / polyaxon

AI Infra / AI Orchestration / AI Control Plane

• deep-learning
• machine-learning
• artificial-intelligence
• data-science
• reinforcement-learning
• kubernetes
• +14 more

84.awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

• dataquality
• spark
• unit-testing
• scala

85.apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

• sql
• spark
• hive
• pyspark
• livy
• linkis
• +14 more

86.alibaba / pipcook

Machine learning platform for Web developers

• js
• machine-learning
• tensorflow
• pipeline

87.janoodleFTW / timy-messenger

Timy - open source mobile app for groups to communicate and organize themselves. Built with flutter.

• flutter
• swift
• kotlin
• messenger
• messaging
• firebase
• +3 more

88.FoundatioFx / Foundatio

Pluggable foundation blocks for building distributed apps.

• azure
• c-sharp
• lock
• message-bus
• storage
• redis
• +14 more

89.asc-lab / dotnetcore-microservices-poc

Very simplified insurance sales system made in a microservices architecture using .NET Core

• netcore
• dotnet-core
• microservices
• insurance
• ddd
• cqrs
• +11 more

90.Netflix / genie

Distributed Big Data Orchestration Service

• big-data
• bigdata
• orchestration
• configuration
• configuration-management
• java
• +7 more

91.apache / eventmesh

EventMesh is a new generation serverless event middleware for building distributed event-driven applications.

• esb
• event-driven
• event-mesh
• event-governance
• cloud-native
• serverless
• +11 more

92.WallarooLabs / wally

Distributed Stream Processing

• stream-processing
• stream-processor
• stream-processing-engine
• pony-language
• python
• linux
• +4 more

93.actionhero / node-resque

Node.js Background jobs backed by redis.

• resque
• resque-plugin
• resque-workers
• background-jobs
• queue
• nodejs
• +1 more

94.aio-libs / aiokafka

asyncio client for kafka

• asyncio
• kafka
• kafka-client

95.arkflow-rs / arkflow

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream
• sql
• datafusion
• +14 more

96.graphframes / graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe
• dataframes
• graphs
• +8 more

97.nanomsg / mangos

mangos is a pure Golang implementation of nanomsg's "Scalablilty Protocols"

• nanomsg
• message
• queue
• nng
• pubsub
• golang-network

98.mrpowers-io / quinn

pyspark methods to enhance developer productivity 📣 👯 🎉

• pyspark
• apache-spark

99.nats-io / nats.java

Java client for NATS

• nats
• java
• client
• messaging
• messaging-library
• middleware

100.capitalone / datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

• python
• pandas
• spark
• data
• data-science
• compare
• +8 more