Trending repositories

1.google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

• mediapipe
• c-plus-plus
• computer-vision
• deep-learning
• android
• video-processing
• +11 more

2.kestra-io / kestra

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code
• automation
• devops
• +14 more

3.trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

• java
• presto
• hive
• hadoop
• big-data
• sql
• +13 more

4.apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python
• scheduler
• workflow
• +14 more

5.ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning
• reinforcement-learning
• deep-learning
• +14 more

6.apache / spark

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java
• big-data
• jdbc
• +2 more

7.nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.

• go
• golang
• messaging
• message-bus
• message-queue
• cloud-native
• +9 more

8.cocoindex-io / cocoindex

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python
• +14 more

9.juicedata / juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

• filesystem
• cloud-native
• golang
• redis
• distributed-systems
• storage
• +6 more

10.Unstructured-IO / unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp
• ocr
• information-retrieval
• +14 more

11.debezium / debezium

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium
• cdc
• database
• +5 more

12.dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake
• data-loading
• data-warehouse
• +4 more

13.apache / doris

Apache Doris is a real-time analytics and hybrid search database for AI agents.

• olap
• database
• hudi
• iceberg
• real-time
• sql
• +12 more

14.dbt-labs / dbt

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

• dbt-viewpoint
• slack
• pypa
• data-modeling
• business-intelligence
• analytics
• +1 more

15.dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science
• workflow-automation
• python
• +9 more

16.openimsdk / open-im-server

IM Chat OpenClaw

• im
• chat
• messaging
• messenger
• openclaw

17.PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science
• workflow-engine
• prefect
• +8 more

18.StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

• database
• olap
• sql
• analytics
• big-data
• realtime-database
• +14 more

19.centrifugal / centrifugo

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

• websocket
• redis
• real-time
• messaging
• scalability
• eventsource
• +14 more

20.apache / seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time
• apache
• batch
• +8 more

21.apache / iggy

Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed

• messaging
• quic
• rust
• streaming
• http
• iggy
• +2 more

22.quickwit-oss / quickwit

Cloud-native OSS search engine for observability

• rust
• log-management
• logs
• tantivy
• cloud-native
• open-source
• +4 more

23.evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning
• model-monitoring
• html-report
• +8 more

24.kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

• kubernetes
• serverless
• autoscaling
• event-driven
• keda
• hacktoberfest

25.slothflowlabs / duckle

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

• data-engineering
• data-integration
• data-pipeline
• data-quality
• duckdb
• elt
• +14 more

26.catboost / catboost

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

• machine-learning
• decision-trees
• gradient-boosting
• gbm
• gbdt
• python
• +12 more

27.apache / flink

Apache Flink

• scala
• java
• big-data
• flink
• python
• sql

28.apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler
• job-scheduler
• cloud-native
• +5 more

29.vesoft-inc / nebula

A distributed, fast open-source graph database featuring horizontal scalability and high availability

• graph-database
• distributed
• database
• graphdb
• raft
• cpp
• +11 more

30.aeron-io / aeron

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

• messaging
• java
• c-plus-plus
• ipc
• multicast-streams
• c

31.apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query
• stratosphere
• metalake
• +5 more

32.andkret / Cookbook

The Data Engineering Cookbook

• data-engineer
• data-engineering
• big-data
• best-practices
• cookbook

33.macrozheng / mall-swarm

mall-swarm是一套微服务商城系统，采用了 Spring Cloud Alibaba、Spring Boot 3.5、Sa-Token、MyBatis、Elasticsearch、Docker、Kubernetes等核心技术，同时提供了基于Vue的管理后台方便快速搭建系统。mall-swarm在电商业务的基础集成了注册中心、配置中心、监控中心、网关等系统功能。

• springcloud
• springboot
• docker
• java
• mysql
• elasticsearch
• +8 more

34.apache / hadoop

Apache Hadoop

• hadoop

35.ibis-project / ibis

the portable Python dataframe library

• python
• impala
• pandas
• database
• clickhouse
• postgresql
• +12 more

36.apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap
• python
• query-engine
• +3 more

37.bruin-data / bruin

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines
• python
• snowflake
• +5 more

38.fivetran / great_expectations

Always know what to expect from your data.

• pipeline-tests
• dataquality
• datacleaning
• datacleaner
• data-science
• data-profiling
• +14 more

39.panjf2000 / gnet

🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go.

• event-driven
• event-loop
• networking
• non-blocking
• epoll
• kqueue
• +10 more

40.cython / cython

The most widely used Python to C compiler

• python
• cython
• cpython
• cpython-extensions
• c
• cpp
• +2 more

41.delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics
• delta-lake

42.apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam
• batch
• golang
• +2 more

43.apache / couchdb

Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability

• content
• network-server
• http
• cloud
• erlang
• javascript
• +4 more

44.apache / camel

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices
• cloud-native
• data-transformation
• +10 more

45.ArroyoSystems / arroyo

Distributed stream processing engine in Rust

• data
• dev-tools
• infrastructure
• kafka
• rust
• sql
• +3 more

46.risingwavelabs / risingwave

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql
• kafka
• materialized-view
• +5 more

47.PRQL / prql

PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement

• data
• pipeline
• sql

48.arkime / arkime

Arkime is an open source, large scale, full packet capturing, indexing, and database system.

• network-monitoring
• pcap
• packet-capture
• nsm
• javascript
• c
• +2 more

49.ag2ai / faststream

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

• asyncio
• kafka
• python
• rabbitmq
• asyncapi
• distributed-systems
• +7 more

50.clidey / whodb

Where data access meets operational intelligence

• mysql
• postgresql
• ai-agent
• data-catalog
• data-governance
• data-integration
• +14 more

51.bytewax / bytewax

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering
• data-processing
• data-science
• +3 more

52.alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件

53.apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift
• jdbc
• spark-sql
• +4 more

54.timeplus-io / proton

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda
• single-binary
• stream-processing
• +10 more

55.microcks / microcks

The open source, cloud native tool for API Mocking and Testing. Microcks is a Cloud Native Computing Foundation incubating project 🚀

• api
• testing
• kubernetes
• api-testing
• event-driven
• graphql
• +14 more

56.apache / rocketmq

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.

• rocketmq
• cloud-native
• eventing
• java
• messaging
• streaming
• +1 more

57.h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

• h2o
• machine-learning
• data-science
• deep-learning
• big-data
• ensemble-learning
• +14 more

58.feast-dev / feast

The Open Source Feature Store for AI/ML

• machine-learning
• features
• ml
• big-data
• feature-store
• python
• +4 more

59.nats-io / nats.go

Golang client for NATS, the cloud native messaging system.

• go
• golang
• nats
• microservices
• microservices-architecture
• pub-sub
• +3 more

60.apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.

• hudi
• apachehudi
• datalake
• bigdata
• apachespark
• incremental-processing
• +3 more

61.asyncapi / spec

The AsyncAPI specification allows you to create machine-readable definitions of your asynchronous APIs.

• asyncapi
• reactive
• specification
• websockets
• amqp
• mqtt
• +2 more

62.krakend / krakend-ce

KrakenD Community Edition: High-performance, stateless, declarative, API Gateway written in Go.

• api-gateway
• microservices
• backend-services
• reverse-proxy
• cloud-native
• microservice
• +8 more

63.apache / hamilton

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering
• dataframe
• etl
• +13 more

64.eclipse-iceoryx / iceoryx

Eclipse iceoryx™ - true zero-copy inter-process-communication

• iceoryx
• eclipse
• ipc
• inter-process-communication
• cpp
• zero-copy
• +10 more

65.lf-edge / ekuiper

Lightweight data stream processing engine for IoT edge

• edge
• sql
• iot
• edge-analytics
• edge-streaming
• edge-computing
• +2 more

66.OpenHFT / Chronicle-Queue

Micro second messaging that stores everything to disk

• chronicle
• queue
• java
• performance
• low-latency
• persistance

67.apache / hugegraph

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)

• database
• graph-database
• graph
• gremlin
• big-data
• graphdb

68.softwaremill / elasticmq

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.

• scala
• sqs-interface
• elasticmq
• amazon-sqs
• aws
• aws-sqs
• +1 more

69.numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

• kubernetes
• stream-processing
• data-processing
• k8s
• pipeline
• map-reduce
• +1 more

70.pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)

• cdc
• tidb
• mysql
• kafka
• ticdc
• dm

71.apache / predictionioArchived

PredictionIO, a machine learning server for developers and ML engineers.

• scala
• big-data
• predictionio

72.MassTransit / MassTransit

Distributed Application Framework for .NET

• masstransit
• rabbitmq
• c-sharp
• azure-service-bus
• messaging
• dotnet
• +2 more

73.opengoofy / hippo4j

📌 异步线程池框架，支持线程池动态变更&监控&报警，无需修改代码轻松引入。Asynchronous thread pool framework, support Thread Pool Dynamic Change & monitoring & Alarm, no need to modify the code easily introduced.

• java
• thread-pool
• dynamic
• monitor
• dynamic-threadpool
• resizable
• +4 more

74.apache / ignite

Apache Ignite

• distributed-sql-database
• iot
• osgi
• network-client
• ignite
• data-management-platform
• +9 more

75.meetfranz / franz

Franz is a free messaging app for services like WhatsApp, Slack, Messenger and many more.

• messaging
• electron
• franz
• messenger
• whatsapp
• slack
• +3 more

76.build-trust / ockam

Orchestrate end-to-end encryption, cryptographic identities, mutual authentication, and authorization policies between distributed applications – at massive scale.

• messaging
• authentication
• key-management
• identity
• trust
• kafka
• +12 more

77.RoaringBitmap / RoaringBitmap

A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others

• java
• bitset
• roaring-bitmaps
• roaringbitmap
• druid
• spark
• +1 more

78.vernemq / vernemq

A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases. The VerneMQ mission is active & the project maintained. Thank you for your support!

• mqtt
• m2m
• scalable
• broker
• erlang
• distributed
• +13 more

79.apache / avro

Apache Avro is a data serialization system.

• csharp
• ruby
• cplusplus
• python
• php
• java
• +5 more

80.WeBankFinTech / DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

• workflow
• governance
• azkaban
• davinci
• linkis
• spark
• +14 more

81.bloomberg / blazingmq

A modern high-performance open source message queuing system

• distributed-systems
• message-queue
• messaging
• middleware
• queue
• queues
• +1 more

82.absmach / magistrala

IoT Platform Framework

• golang
• iot-platform
• edge-computing
• distributed-systems
• iot-cloud
• iot-gateway
• +14 more

83.elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.

• notebooks
• notebook-jupyter
• jupyterlab
• jupyterlab-extensions
• ai
• machine-learning
• +14 more

84.matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS

• aws
• cloud
• security
• big-data
• serverless
• apache-iceberg
• +14 more

85.quixio / quix-streams

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering
• data-science
• machine-learning
• +8 more

86.projectnessie / nessie

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

• data
• spark
• java
• iceberg
• aws-lambda
• git

87.apache / cloudberry

One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.

• database
• data-analysis
• data-warehouse
• olap
• postgresql
• sql
• +8 more

88.halaxa / json-machine

Efficient, easy-to-use, and fast PHP JSON stream parser

• php
• stream-processing
• json-parser
• json-iterator
• json-stream
• parsing

89.PatMartin / Dex

Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.

• data-science
• data-visualization
• visualization
• data-analysis
• data-mining
• java
• +7 more

90.spring-cloud / spring-cloud-stream

Framework for building Event-Driven Microservices

• microservices
• messaging
• event-driven-microservices
• stream-processing
• reactive
• cloud-native

91.thingsboard / tbmq

The ultimate distributed MQTT broker. Handles 100M+ connections and 10M msg/sec with ease. Built on Kafka to provide industrial-grade persistence and eliminate data loss.

• broker
• iot
• java
• kafka
• mqtt
• mqtt-broker
• +5 more

92.apache / streampipes

Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

• iot
• analytics
• edge
• stream-processing
• iiot
• self-service
• +6 more

93.samber / ro

🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications

• event-driven
• generics
• go
• golang
• pipe
• pipeline
• +11 more

94.zhisheng17 / flink-learning

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

• flink
• kafka
• elasticsearch
• spark
• redis
• mysql
• +9 more

95.deeplearning4j / deeplearning4j

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...

• java
• gpu
• deeplearning
• neural-nets
• deeplearning4j
• dl4j
• +9 more

96.provectus / kafka-ui

Open-Source Web UI for Apache Kafka Management

• kafka-ui
• kafka-brokers
• kafka
• kafka-streams
• kafka-client
• opensource
• +12 more

97.snowplow / snowplow

The leader in Customer Data Infrastructure

• snowplow
• analytics
• data
• data-pipeline
• data-collection
• product-analytics
• +3 more

98.confluentinc / confluent-kafka-go

Confluent's Apache Kafka Golang client

• confluent
• golang
• golang-library
• golang-bindings
• librdkafka
• kafka-client
• +2 more

99.awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

• dataquality
• spark
• unit-testing
• scala

100.apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

• sql
• spark
• hive
• pyspark
• livy
• linkis
• +14 more