Trending repositories

1.stepll2026 / OntiCards

OntiCards 是面向企业的 AI 数据中枢（AI Data Hub），围绕业务数据库提供一站式能力：数据源连接、元数据理解、自然语言查询、跨源融合、数据质检、查询审计与数据资产管理等

2.google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

• mediapipe
• c-plus-plus
• computer-vision
• deep-learning
• android
• video-processing
• +11 more

3.kestra-io / kestra

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code
• automation
• devops
• +14 more

4.ModernRelay / omnigraph

Object-storage native graph database with branching and typed ontology

• apache-arrow
• datafusion
• graph-database
• knowledge-graph
• lakehouse
• lance
• +5 more

5.dagucloud / dagu

Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.

• cron
• workflow-engine
• workflow-scheduler
• workflow-orchestration
• devops
• data-pipeline
• +14 more

6.open-metadata / OpenMetadata

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

• metadata
• datadiscovery
• dataquality
• data-profiling
• metadata-management
• data-catalog
• +14 more

7.apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python
• scheduler
• workflow
• +14 more

8.ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning
• reinforcement-learning
• deep-learning
• +14 more

9.apache / spark

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java
• big-data
• jdbc
• +2 more

10.nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.

• go
• golang
• messaging
• message-bus
• message-queue
• cloud-native
• +9 more

11.cocoindex-io / cocoindex

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python
• +14 more

12.Unstructured-IO / unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp
• ocr
• information-retrieval
• +14 more

13.restatedev / restate

Restate is the platform for building resilient applications that tolerate all infrastructure faults w/o the need for a PhD.

• distributed-systems
• async-await
• durable-execution
• microservices
• consistent-state
• reliable-communication
• +4 more

14.airbytehq / airbyte

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering
• java
• python
• +14 more

15.hugues31 / duckpad

The data transformation tool the open-source community deserves.

• data-engineering
• duckdb

16.debezium / debezium

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium
• cdc
• database
• +5 more

17.dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake
• data-loading
• data-warehouse
• +4 more

18.projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.

• http
• bugbounty
• ssl-certificate
• pipeline
• pentest-tool
• cybersecurity
• +4 more

19.hatchet-dev / hatchet

🪓 An orchestration engine for background tasks, AI agents, and durable workflows

• event-driven
• golang
• workflow-engine
• distributed
• distributed-systems
• fastapi
• +7 more

20.riverqueue / river

The polyglot queue: Fast and reliable background jobs in Go, Java, JS, Ruby, and Rust on Postgres/SQLite.

• background-jobs
• go
• golang
• postgres
• postgresql
• queue
• +5 more

21.dbt-labs / dbt

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

• dbt-viewpoint
• slack
• pypa
• data-modeling
• business-intelligence
• analytics
• +1 more

22.timgit / pg-boss

Queueing jobs in Postgres from Node.js like a boss

• postgresql
• postgres
• queue
• nodejs
• node
• job

23.dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science
• workflow-automation
• python
• +9 more

24.eclipse-mosquitto / mosquitto

Eclipse Mosquitto - An open source MQTT broker

• mqtt
• broker
• mosquitto
• eclipse-iot
• iot
• messaging
• +3 more

25.growthbook / growthbook

Open Source Feature Flags, Experimentation, and Product Analytics

• abtesting
• statistics
• abtest
• experimentation
• split-testing
• snowflake
• +14 more

26.thingsboard / thingsboard

All-in-one IoT Platform - Device management, data collection, processing and visualization.

• visualization
• iot
• iot-platform
• mqtt
• middleware
• thingsboard
• +12 more

27.openimsdk / open-im-server

IM Chat OpenClaw

• im
• chat
• messaging
• messenger
• openclaw

28.PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science
• workflow-engine
• prefect
• +8 more

29.StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

• database
• olap
• sql
• analytics
• big-data
• realtime-database
• +14 more

30.centrifugal / centrifugo

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

• websocket
• redis
• real-time
• messaging
• scalability
• eventsource
• +14 more

31.apache / seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time
• apache
• batch
• +8 more

32.tobymao / sqlglot

Python SQL Parser and Transpiler

• transpiler
• sql
• python
• parser
• optimizer
• bigquery
• +14 more

33.taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL

• background-jobs
• nodejs
• queue
• redis
• typescript
• python
• +3 more

34.inngest / inngest

The leading workflow orchestration platform. Run stateful step functions and AI workflows on serverless, servers, or the edge.

• cli
• event-driven
• serverless-functions
• queues
• serverless
• event-driven-architecture
• +2 more

35.apache / iggy

Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed

• messaging
• quic
• rust
• streaming
• http
• iggy
• +2 more

36.emqx / emqx

The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles

• mqtt
• iot
• mqtt-broker
• erlang
• iot-middleware
• broker
• +14 more

37.quickwit-oss / quickwit

Cloud-native OSS search engine for observability

• rust
• log-management
• logs
• tantivy
• cloud-native
• open-source
• +4 more

38.evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning
• model-monitoring
• html-report
• +8 more

39.evidence-dev / evidence

Business intelligence as code: build fast, interactive data visualizations in SQL and markdown

• analytics
• sql
• business-intelligence
• data-visualization
• dbt
• duckdb
• +8 more

40.redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

• kafka
• cpp
• kubernetes
• go
• streaming
• redpanda
• +7 more

41.lightdash / lightdash

Agentic BI. Analytics at the speed of code ⚡️

• dbt
• data-visualization
• data-analytics
• business-intelligence

42.kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

• kubernetes
• serverless
• autoscaling
• event-driven
• keda
• hacktoberfest

43.slothflowlabs / duckle

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

• data-engineering
• data-integration
• data-pipeline
• data-quality
• duckdb
• elt
• +14 more

44.zeromq / libzmq

ZeroMQ core engine in C++, implements ZMTP/3.1

• zeromq
• zmq
• messaging
• concurrency
• pubsub
• pushpull
• +5 more

45.pgmq / pgmq

A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.

• message-queue

46.catboost / catboost

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

• machine-learning
• decision-trees
• gradient-boosting
• gbm
• gbdt
• python
• +12 more

47.apache / flink

Apache Flink

• scala
• java
• big-data
• flink
• python
• sql

48.apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler
• job-scheduler
• cloud-native
• +5 more

49.vesoft-inc / nebula

A distributed, fast open-source graph database featuring horizontal scalability and high availability

• graph-database
• distributed
• database
• graphdb
• raft
• cpp
• +11 more

50.PipedreamHQ / pipedream

Connect APIs, remarkably fast. Free for developers.

• eventsourcing
• event-sourcing
• automation
• workflow
• workflows
• apis
• +13 more

51.aeron-io / aeron

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

• messaging
• java
• c-plus-plus
• ipc
• multicast-streams
• c

52.svix / svix-webhooks

The open source and enterprise-ready webhooks service 🦀

• webhooks
• rust
• webhook-server
• webhook-service
• webhook
• webhooks-server
• +13 more

53.apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query
• stratosphere
• metalake
• +5 more

54.madd86 / awesome-system-design

A curated list of awesome System Design (A.K.A. Distributed Systems) resources.

• relational-database
• message-broker
• hadoop-ecosystem
• nosql
• interview
• distributed-systems
• +3 more

55.ThreeDotsLabs / watermill

Building event-driven applications the easy way in Go.

• watermill
• event-driven
• event-sourcing
• sagas
• events
• stream-processing
• +7 more

56.apache / hadoop

Apache Hadoop

• hadoop

57.ibis-project / ibis

the portable Python dataframe library

• python
• impala
• pandas
• database
• clickhouse
• postgresql
• +12 more

58.bruin-data / ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

• bigquery
• copy-database
• data-ingestion
• data-integration
• data-pipeline
• duckdb
• +4 more

59.apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap
• python
• query-engine
• +3 more

60.bruin-data / bruin

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines
• python
• snowflake
• +5 more

61.alibaba / DataX

DataX是阿里云DataWorks数据集成的开源版本。

62.tinode / chat

Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots

• chat
• chat-application
• chatapp
• go
• golang
• instant-messaging
• +14 more

63.heynickc / awesome-ddd

A curated list of Domain-Driven Design (DDD), Command Query Responsibility Segregation (CQRS), Event Sourcing, and Event Storming resources

• awesome
• awesome-list
• list

64.fivetran / great_expectations

Always know what to expect from your data.

• pipeline-tests
• dataquality
• datacleaning
• datacleaner
• data-science
• data-profiling
• +14 more

65.panjf2000 / gnet

🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go.

• event-driven
• event-loop
• networking
• non-blocking
• epoll
• kqueue
• +10 more

66.cython / cython

The most widely used Python to C compiler

• python
• cython
• cpython
• cpython-extensions
• c
• cpp
• +2 more

67.delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics
• delta-lake

68.apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam
• batch
• golang
• +2 more

69.apache / couchdb

Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability

• content
• network-server
• http
• cloud
• erlang
• javascript
• +4 more

70.apache / camel

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices
• cloud-native
• data-transformation
• +10 more

71.kurrent-io / KurrentDB

KurrentDB is a database that's engineered for modern software applications and event-driven architectures. Its event-native design simplifies data modeling and preserves data integrity while the integrated streaming engine solves distributed messaging challenges and ensures data consistency.

• cqrs
• eventsourcing
• eventstore
• event-sourcing
• event-store
• database

72.ArroyoSystems / arroyo

Distributed stream processing engine in Rust

• data
• dev-tools
• infrastructure
• kafka
• rust
• sql
• +3 more

73.rudderlabs / rudder-server

Privacy and Security focused Segment-alternative, in Golang and React

• privacy
• warehouse-management
• data-warehouse
• customer-data
• customer-data-pipeline
• customer-data-platform
• +14 more

74.confluentinc / librdkafka

The Apache Kafka C/C++ library

• kafka
• kafka-consumer
• apache-kafka
• high-performance
• c
• librdkafka
• +3 more

75.risingwavelabs / risingwave

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql
• kafka
• materialized-view
• +5 more

76.ergo-services / ergo

An actor-based Framework with network transparency for creating event-driven architecture in Golang. Inspired by Erlang. Zero dependencies.

• erlang
• golang
• elixir
• supervisor
• otp
• otp-applications
• +14 more

77.apecloud / kubeblocks

KubeBlocks is a Kubernetes Operator designed to manage a variety of databases and streaming systems, including MySQL, PostgreSQL, MongoDB, Redis, RabbitMQ, RocketMQ, and more, within Kubernetes environments.

• mysql-operator
• mongodb-operator
• postgresql-operator
• rabbitmq-operator
• redis-operator
• databases
• +13 more

78.emirpasic / gods

GoDS (Go Data Structures) - Sets, Lists, Stacks, Maps, Trees, Queues, and much more

• go
• golang
• data-structure
• map
• tree
• set
• +10 more

79.PRQL / prql

PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement

• data
• pipeline
• sql

80.arkime / arkime

Arkime is an open source, large scale, full packet capturing, indexing, and database system.

• network-monitoring
• pcap
• packet-capture
• nsm
• javascript
• c
• +2 more

81.dotnetcore / CAP

Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern

• kafka
• rabbitmq
• distributed-transactions
• outbox-pattern
• eventbus
• microservices
• +1 more

82.hazelcast / hazelcast

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

• java
• hazelcast
• in-memory
• big-data
• scalability
• distributed
• +9 more

83.ag2ai / faststream

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

• asyncio
• kafka
• python
• rabbitmq
• asyncapi
• distributed-systems
• +7 more

84.clidey / whodb

Where data access meets operational intelligence

• mysql
• postgresql
• ai-agent
• data-catalog
• data-governance
• data-integration
• +14 more

85.graphile / worker

High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)

• postgresql
• node
• worker
• job-queue
• graphile
• nodejs
• +2 more

86.bytewax / bytewax

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering
• data-processing
• data-science
• +3 more

87.containrrr / shoutrrr

Notification library for gophers and their furry friends.

• notifications
• golang
• go
• messaging
• integrations
• hacktoberfest

88.alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件

89.rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins

• rabbitmq
• amqp
• messaging
• amqp-0-9-1
• amqp1-0
• mqtt
• +4 more

90.dask / dask

Parallel computing with task scheduling

• dask
• python
• pydata
• numpy
• pandas
• scikit-learn
• +1 more

91.RichardKnop / machinery

Machinery is an asynchronous task queue/job queue based on distributed message passing.

• go
• golang
• task
• task-scheduler
• queue
• amqp
• +5 more

92.mahmoud / boltons

🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

• python
• utilities
• cache
• data-structures
• statistics
• data-science
• +5 more

93.sindresorhus / p-queue

Promise queue with concurrency control

• queue
• queue-data-stucture
• async-functions
• async-queue
• promise-queue
• promise
• +2 more

94.onetimesecret / onetimesecret

Keep passwords and other sensitive information out of your chat logs and inboxes.

• privacy
• onetime
• secrets
• chat
• email
• messaging
• +1 more

95.meltano / meltano

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

• dataops
• dataops-platform
• elt
• open-source
• opensource
• data
• +14 more

96.nanomq / nanomq

An ultra-lightweight and blazing-fast Messaging Bus/MQTT Broker for Edge & SDV

• mqtt
• mqtt-broker
• iot-edge
• nng
• aio
• actor-model
• +14 more

97.apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift
• jdbc
• spark-sql
• +4 more

98.timeplus-io / proton

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda
• single-binary
• stream-processing
• +10 more

99.microcks / microcks

The open source, cloud native tool for API Mocking and Testing. Microcks is a Cloud Native Computing Foundation incubating project 🚀

• api
• testing
• kubernetes
• api-testing
• event-driven
• graphql
• +14 more

100.Desbordante / desbordante-core

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

• data-analytics
• data-cleaning
• data-cleansing
• data-engineering
• data-exploration
• data-mining
• +14 more