Trending repositories

1.OpenDCAI / DataFlow

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

• data
• data-cleaning
• data-pipelines
• data-processing
• data-science
• data-synthesis
• +7 more

2.kestra-io / kestra

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code
• automation
• devops
• +14 more

3.cocoindex-io / cocoindex

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python
• +14 more

4.google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

• mediapipe
• c-plus-plus
• computer-vision
• deep-learning
• android
• video-processing
• +11 more

5.apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python
• scheduler
• workflow
• +14 more

6.open-metadata / OpenMetadata

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

• metadata
• datadiscovery
• dataquality
• data-profiling
• metadata-management
• data-catalog
• +14 more

7.ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning
• reinforcement-learning
• deep-learning
• +14 more

8.PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science
• workflow-engine
• prefect
• +8 more

9.apache / kafka

Apache Kafka - A distributed event streaming platform

• scala
• kafka
• java
• streaming

10.dbt-labs / dbt

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

• dbt-viewpoint
• slack
• pypa
• data-modeling
• business-intelligence
• analytics
• +1 more

11.apache / spark

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java
• big-data
• jdbc
• +2 more

12.nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.

• go
• golang
• messaging
• message-bus
• message-queue
• cloud-native
• +9 more

13.AutoMQ / automq

Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

• kafka
• diskless
• automq
• serverless

14.Unstructured-IO / unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp
• ocr
• information-retrieval
• +14 more

15.apache / iggy

Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed

• messaging
• quic
• rust
• streaming
• http
• iggy
• +2 more

16.dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science
• workflow-automation
• python
• +9 more

17.dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake
• data-loading
• data-warehouse
• +4 more

18.centrifugal / centrifugo

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

• websocket
• redis
• real-time
• messaging
• scalability
• eventsource
• +14 more

19.debezium / debezium

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium
• cdc
• database
• +5 more

20.apache / iceberg

Apache Iceberg

• iceberg
• apache
• hacktoberfest

21.evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning
• model-monitoring
• html-report
• +8 more

22.apache / flink

Apache Flink

• scala
• java
• big-data
• flink
• python
• sql

23.openimsdk / open-im-server

IM Chat OpenClaw

• im
• chat
• messaging
• messenger
• openclaw

24.apache / seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time
• apache
• batch
• +8 more

25.fivetran / great_expectations

Always know what to expect from your data.

• pipeline-tests
• dataquality
• datacleaning
• datacleaner
• data-science
• data-profiling
• +14 more

26.apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query
• stratosphere
• metalake
• +5 more

27.risingwavelabs / risingwave

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql
• kafka
• materialized-view
• +5 more

28.numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

• kubernetes
• stream-processing
• data-processing
• k8s
• pipeline
• map-reduce
• +1 more

29.feast-dev / feast

The Open Source Feature Store for AI/ML

• machine-learning
• features
• ml
• big-data
• feature-store
• python
• +4 more

30.aeron-io / aeron

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

• messaging
• java
• c-plus-plus
• ipc
• multicast-streams
• c

31.apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler
• job-scheduler
• cloud-native
• +5 more

32.apache / rocketmq

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.

• rocketmq
• cloud-native
• eventing
• java
• messaging
• streaming
• +1 more

33.provectus / kafka-ui

Open-Source Web UI for Apache Kafka Management

• kafka-ui
• kafka-brokers
• kafka
• kafka-streams
• kafka-client
• opensource
• +12 more

34.delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics
• delta-lake

35.cleanlab / cleanlab

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

• weak-supervision
• data-cleaning
• data-quality
• noisy-labels
• data-centric-ai
• out-of-distribution-detection
• +14 more

36.Netflix / metaflow

Build, Manage and Deploy AI/ML Systems

• machine-learning
• model-management
• ai
• ml
• ml-platform
• ml-infrastructure
• +14 more

37.clidey / whodb

Where data access meets operational intelligence

• mysql
• postgresql
• ai-agent
• data-catalog
• data-governance
• data-integration
• +14 more

38.ag2ai / faststream

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

• asyncio
• kafka
• python
• rabbitmq
• asyncapi
• distributed-systems
• +7 more

39.apache / hadoop

Apache Hadoop

• hadoop

40.nats-io / nats.go

Golang client for NATS, the cloud native messaging system.

• go
• golang
• nats
• microservices
• microservices-architecture
• pub-sub
• +3 more

41.apache / cloudberry

One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.

• database
• data-analysis
• data-warehouse
• olap
• postgresql
• sql
• +8 more

42.bruin-data / bruin

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines
• python
• snowflake
• +5 more

43.alibaba / canal

TranslatedAlibaba's MySQL binlog incremental subscription & consumption component

44.arkime / arkime

Arkime is an open source, large scale, full packet capturing, indexing, and database system.

• network-monitoring
• pcap
• packet-capture
• nsm
• javascript
• c
• +2 more

45.apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.

• hudi
• apachehudi
• datalake
• bigdata
• apachespark
• incremental-processing
• +3 more

46.apache / camel

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices
• cloud-native
• data-transformation
• +10 more

47.apache / pulsar

Apache Pulsar - distributed pub-sub messaging system

• pulsar
• pubsub
• messaging
• streaming
• queuing
• event-streaming

48.ArroyoSystems / arroyo

Distributed stream processing engine in Rust

• data
• dev-tools
• infrastructure
• kafka
• rust
• sql
• +3 more

49.apache / shardingsphere

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql
• shard
• database-cluster
• +10 more

50.ibis-project / ibis

the portable Python dataframe library

• python
• impala
• pandas
• database
• clickhouse
• postgresql
• +12 more

51.asyncapi / spec

The AsyncAPI specification allows you to create machine-readable definitions of your asynchronous APIs.

• asyncapi
• reactive
• specification
• websockets
• amqp
• mqtt
• +2 more

52.microcks / microcks

The open source, cloud native tool for API Mocking and Testing. Microcks is a Cloud Native Computing Foundation incubating project 🚀

• api
• testing
• kubernetes
• api-testing
• event-driven
• graphql
• +14 more

53.apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam
• batch
• golang
• +2 more

54.apache / couchdb

Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability

• content
• network-server
• http
• cloud
• erlang
• javascript
• +4 more

55.apache / hamilton

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering
• dataframe
• etl
• +13 more

56.eclipse-iceoryx / iceoryx

Eclipse iceoryx™ - true zero-copy inter-process-communication

• iceoryx
• eclipse
• ipc
• inter-process-communication
• cpp
• zero-copy
• +10 more

57.apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap
• python
• query-engine
• +3 more

58.spotify / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

• python
• luigi
• orchestration-framework
• scheduling
• hadoop

59.apache / hugegraph

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)

• database
• graph-database
• graph
• gremlin
• big-data
• graphdb

60.softwaremill / elasticmq

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.

• scala
• sqs-interface
• elasticmq
• amazon-sqs
• aws
• aws-sqs
• +1 more

61.cloudamqp / lavinmq

Ultra quick message queue and streaming server

• message-queue
• server
• amqp
• background-jobs
• job-queue
• message-broker
• +3 more

62.MassTransit / MassTransit

Distributed Application Framework for .NET

• masstransit
• rabbitmq
• c-sharp
• azure-service-bus
• messaging
• dotnet
• +2 more

63.RoaringBitmap / RoaringBitmap

A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others

• java
• bitset
• roaring-bitmaps
• roaringbitmap
• druid
• spark
• +1 more

64.fluvio-community / fluvio

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

• cloud-native
• streaming
• rust
• real-time
• serverless
• stateful
• +13 more

65.OpenHFT / Chronicle-Queue

Micro second messaging that stores everything to disk

• chronicle
• queue
• java
• performance
• low-latency
• persistance

66.krakend / krakend-ce

KrakenD Community Edition: High-performance, stateless, declarative, API Gateway written in Go.

• api-gateway
• microservices
• backend-services
• reverse-proxy
• cloud-native
• microservice
• +8 more

67.timeplus-io / proton

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda
• single-binary
• stream-processing
• +10 more

68.h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

• h2o
• machine-learning
• data-science
• deep-learning
• big-data
• ensemble-learning
• +14 more

69.beava-dev / beava

Real-time decision features without streaming infra. Turn live events into product reflexes — no Kafka, no Flink, no feature store.

• analytics
• feature-store
• fraud-detection
• python
• real-time
• rust
• +4 more

70.bytewax / bytewax

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering
• data-processing
• data-science
• +3 more

71.reductstore / reductstore

High-performance, time-indexed object storage for robotics and industrial IoT

• blob-storage
• storage-engine
• storage
• historian
• timeseries
• time-series
• +9 more

72.vernemq / vernemq

A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases. The VerneMQ mission is active & the project maintained. Thank you for your support!

• mqtt
• m2m
• scalable
• broker
• erlang
• distributed
• +13 more

73.absmach / magistrala

IoT Platform Framework

• golang
• iot-platform
• edge-computing
• distributed-systems
• iot-cloud
• iot-gateway
• +14 more

74.projectnessie / nessie

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

• data
• spark
• java
• iceberg
• aws-lambda
• git

75.meetfranz / franz

Franz is a free messaging app for services like WhatsApp, Slack, Messenger and many more.

• messaging
• electron
• franz
• messenger
• whatsapp
• slack
• +3 more

76.confluentinc / confluent-kafka-go

Confluent's Apache Kafka Golang client

• confluent
• golang
• golang-library
• golang-bindings
• librdkafka
• kafka-client
• +2 more

77.lf-edge / ekuiper

Lightweight data stream processing engine for IoT edge

• edge
• sql
• iot
• edge-analytics
• edge-streaming
• edge-computing
• +2 more

78.opengoofy / hippo4j

📌 异步线程池框架，支持线程池动态变更&监控&报警，无需修改代码轻松引入。Asynchronous thread pool framework, support Thread Pool Dynamic Change & monitoring & Alarm, no need to modify the code easily introduced.

• java
• thread-pool
• dynamic
• monitor
• dynamic-threadpool
• resizable
• +4 more

79.apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift
• jdbc
• spark-sql
• +4 more

80.apache / avro

Apache Avro is a data serialization system.

• csharp
• ruby
• cplusplus
• python
• php
• java
• +5 more

81.bloomberg / blazingmq

A modern high-performance open source message queuing system

• distributed-systems
• message-queue
• messaging
• middleware
• queue
• queues
• +1 more

82.awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

• dataquality
• spark
• unit-testing
• scala

83.snowplow / snowplow

The leader in Customer Data Infrastructure

• snowplow
• analytics
• data
• data-pipeline
• data-collection
• product-analytics
• +3 more

84.nats-io / nats.py

Python3 client for NATS

• asyncio
• nats
• python3
• cloud-native
• aio-nats
• aio

85.apache / artemis

Apache Artemis

• java
• apache
• messaging
• broker
• amqp
• amqps
• +7 more

86.samber / ro

🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications

• event-driven
• generics
• go
• golang
• pipe
• pipeline
• +11 more

87.thingsboard / tbmq

The ultimate distributed MQTT broker. Handles 100M+ connections and 10M msg/sec with ease. Built on Kafka to provide industrial-grade persistence and eliminate data loss.

• broker
• iot
• java
• kafka
• mqtt
• mqtt-broker
• +5 more

88.arkflow-rs / arkflow

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream
• sql
• datafusion
• +14 more

89.kroxylicious / kroxylicious

Kroxylicious, the snappy open source proxy for Apache Kafka®

• apache-kafka
• java
• kafka
• proxy

90.apache / ignite

Apache Ignite

• distributed-sql-database
• iot
• osgi
• network-client
• ignite
• data-management-platform
• +9 more

91.JohnSnowLabs / spark-nlp

State of the Art Natural Language Processing

• nlp
• natural-language-processing
• spark
• pyspark
• named-entity-recognition
• sentiment-analysis
• +14 more

92.FluidGroup / swiftui-messaging-ui

For SwiftUI apps - a primitive component to make Chat-UI with stable prepending - no scroll jumps when loading older messages.

• chat
• messaging
• swiftui

93.apache / activemq

Apache ActiveMQ

• activemq
• java
• amqp
• amqps
• apache
• broker
• +5 more

94.graphframes / graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe
• dataframes
• graphs
• +8 more

95.WeBankFinTech / DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

• workflow
• governance
• azkaban
• davinci
• linkis
• spark
• +14 more

96.quixio / quix-streams

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering
• data-science
• machine-learning
• +8 more

97.build-trust / ockam

Orchestrate end-to-end encryption, cryptographic identities, mutual authentication, and authorization policies between distributed applications – at massive scale.

• messaging
• authentication
• key-management
• identity
• trust
• kafka
• +12 more

98.matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS

• aws
• cloud
• security
• big-data
• serverless
• apache-iceberg
• +14 more

99.apache / streampipes

Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

• iot
• analytics
• edge
• stream-processing
• iiot
• self-service
• +6 more

100.yahoo / CMAK

CMAK is a tool for managing Apache Kafka clusters

• kafka
• scala
• cluster-management
• big-data