Trending repositories

1.simplex-chat / simplex-chat

SimpleX - the first messaging network operating without user identifiers of any kind - 100% private by design! iOS, Android and desktop apps 📱!

• chat
• protocol
• e2ee
• privacy
• haskell
• security
• +3 more

2.kestra-io / kestra

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code
• automation
• devops
• +14 more

3.cocoindex-io / cocoindex

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python
• +14 more

4.google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

• mediapipe
• c-plus-plus
• computer-vision
• deep-learning
• android
• video-processing
• +11 more

5.apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python
• scheduler
• workflow
• +14 more

6.ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning
• reinforcement-learning
• deep-learning
• +14 more

7.PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science
• workflow-engine
• prefect
• +8 more

8.apache / kafka

Apache Kafka - A distributed event streaming platform

• scala
• kafka
• java
• streaming

9.ModernRelay / omnigraph

Object-storage native graph database with branching and typed ontology

• apache-arrow
• datafusion
• graph-database
• knowledge-graph
• lakehouse
• lance
• +5 more

10.dbt-labs / dbt

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

• dbt-viewpoint
• slack
• pypa
• data-modeling
• business-intelligence
• analytics
• +1 more

11.apache / spark

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java
• big-data
• jdbc
• +2 more

12.nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.

• go
• golang
• messaging
• message-bus
• message-queue
• cloud-native
• +9 more

13.AutoMQ / automq

Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

• kafka
• diskless
• automq
• serverless

14.airbytehq / airbyte

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering
• java
• python
• +14 more

15.dagucloud / dagu

Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.

• cron
• workflow-engine
• workflow-scheduler
• workflow-orchestration
• devops
• data-pipeline
• +14 more

16.juicedata / juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

• filesystem
• cloud-native
• golang
• redis
• distributed-systems
• storage
• +6 more

17.hatchet-dev / hatchet

🪓 An orchestration engine for background tasks, AI agents, and durable workflows

• event-driven
• golang
• workflow-engine
• distributed
• distributed-systems
• fastapi
• +7 more

18.Unstructured-IO / unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp
• ocr
• information-retrieval
• +14 more

19.influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

• telegraf
• monitoring
• time-series
• metrics
• golang
• influxdb
• +10 more

20.thingsboard / thingsboard

All-in-one IoT Platform - Device management, data collection, processing and visualization.

• visualization
• iot
• iot-platform
• mqtt
• middleware
• thingsboard
• +12 more

21.apache / iggy

Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed

• messaging
• quic
• rust
• streaming
• http
• iggy
• +2 more

22.dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science
• workflow-automation
• python
• +9 more

23.aklivity / zilla

🦎 A lightweight, streaming-native, multi-protocol gateway for AI, APIs, and real-time data. Route LLM requests across providers, govern MCP tools, and connect Kafka, MQTT, and HTTP through one high-performance engine with shared security and observability.

• kafka
• rest
• server-sent-events
• streaming-api
• grpc
• mqtt
• +9 more

24.apache / doris

Apache Doris is a real-time analytics and hybrid search database for AI agents.

• olap
• database
• hudi
• iceberg
• real-time
• sql
• +12 more

25.growthbook / growthbook

Open Source Feature Flags, Experimentation, and Product Analytics

• abtesting
• statistics
• abtest
• experimentation
• split-testing
• snowflake
• +14 more

26.pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python
• streaming
• machine-learning-algorithms
• +11 more

27.restatedev / restate

Restate is the platform for building resilient applications that tolerate all infrastructure faults w/o the need for a PhD.

• distributed-systems
• async-await
• durable-execution
• microservices
• consistent-state
• reliable-communication
• +4 more

28.evidence-dev / evidence

Business intelligence as code: build fast, interactive data visualizations in SQL and markdown

• analytics
• sql
• business-intelligence
• data-visualization
• dbt
• duckdb
• +8 more

29.taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL

• background-jobs
• nodejs
• queue
• redis
• typescript
• python
• +3 more

30.riverqueue / river

The polyglot queue: Fast and reliable background jobs in Go, Java, JS, Ruby, and Rust on Postgres/SQLite.

• background-jobs
• go
• golang
• postgres
• postgresql
• queue
• +5 more

31.dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake
• data-loading
• data-warehouse
• +4 more

32.projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.

• http
• bugbounty
• ssl-certificate
• pipeline
• pentest-tool
• cybersecurity
• +4 more

33.centrifugal / centrifugo

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

• websocket
• redis
• real-time
• messaging
• scalability
• eventsource
• +14 more

34.emqx / emqx

The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles

• mqtt
• iot
• mqtt-broker
• erlang
• iot-middleware
• broker
• +14 more

35.trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

• java
• presto
• hive
• hadoop
• big-data
• sql
• +13 more

36.inngest / inngest

The leading workflow orchestration platform. Run stateful step functions and AI workflows on serverless, servers, or the edge.

• cli
• event-driven
• serverless-functions
• queues
• serverless
• event-driven-architecture
• +2 more

37.questdb / questdb

QuestDB is a high performance, open-source, time-series database

• time-series
• low-latency
• database
• sql
• questdb
• tsdb
• +14 more

38.debezium / debezium

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium
• cdc
• database
• +5 more

39.redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

• kafka
• cpp
• kubernetes
• go
• streaming
• redpanda
• +7 more

40.quickwit-oss / quickwit

Cloud-native OSS search engine for observability

• rust
• log-management
• logs
• tantivy
• cloud-native
• open-source
• +4 more

41.StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

• database
• olap
• sql
• analytics
• big-data
• realtime-database
• +14 more

42.timgit / pg-boss

Queueing jobs in Postgres from Node.js like a boss

• postgresql
• postgres
• queue
• nodejs
• node
• job

43.evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning
• model-monitoring
• html-report
• +8 more

44.apache / flink

Apache Flink

• scala
• java
• big-data
• flink
• python
• sql

45.bruin-data / ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

• bigquery
• copy-database
• data-ingestion
• data-integration
• data-pipeline
• duckdb
• +4 more

46.tobymao / sqlglot

Python SQL Parser and Transpiler

• transpiler
• sql
• python
• parser
• optimizer
• bigquery
• +14 more

47.madd86 / awesome-system-design

A curated list of awesome System Design (A.K.A. Distributed Systems) resources.

• relational-database
• message-broker
• hadoop-ecosystem
• nosql
• interview
• distributed-systems
• +3 more

48.kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

• kubernetes
• serverless
• autoscaling
• event-driven
• keda
• hacktoberfest

49.pgmq / pgmq

A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.

• message-queue

50.openimsdk / open-im-server

IM Chat OpenClaw

• im
• chat
• messaging
• messenger
• openclaw

51.kgrzybek / modular-monolith-with-ddd

Full Modular Monolith application with Domain-Driven Design approach.

• ddd
• ddd-architecture
• cqrs
• architecture
• rest-api
• dotnet
• +14 more

52.apache / seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time
• apache
• batch
• +8 more

53.eclipse-mosquitto / mosquitto

Eclipse Mosquitto - An open source MQTT broker

• mqtt
• broker
• mosquitto
• eclipse-iot
• iot
• messaging
• +3 more

54.andkret / Cookbook

The Data Engineering Cookbook

• data-engineer
• data-engineering
• big-data
• best-practices
• cookbook

55.fivetran / great_expectations

Always know what to expect from your data.

• pipeline-tests
• dataquality
• datacleaning
• datacleaner
• data-science
• data-profiling
• +14 more

56.PipedreamHQ / pipedream

Connect APIs, remarkably fast. Free for developers.

• eventsourcing
• event-sourcing
• automation
• workflow
• workflows
• apis
• +13 more

57.lightdash / lightdash

Agentic BI. Analytics at the speed of code ⚡️

• dbt
• data-visualization
• data-analytics
• business-intelligence

58.argoproj / argo-workflows

Workflow Engine for Kubernetes

• workflow
• kubernetes
• argo
• dag
• knative
• airflow
• +12 more

59.apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query
• stratosphere
• metalake
• +5 more

60.Avaiga / taipy

Turns Data and AI algorithms into production-ready web applications in no time.

• automation
• data-engineering
• data-ops
• data-visualization
• datascience
• developer-tools
• +14 more

61.risingwavelabs / risingwave

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql
• kafka
• materialized-view
• +5 more

62.oxnr / awesome-bigdata

A curated list of awesome big data frameworks, ressources and other awesomeness.

• awesome
• bigdata
• data-warehouse
• data-analytics
• data-visualization
• data-stream
• +9 more

63.xonsh / xonsh

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

• xonsh
• devops
• iterm2
• data-engineering
• security-automation
• raspberry-pi
• +13 more

64.rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins

• rabbitmq
• amqp
• messaging
• amqp-0-9-1
• amqp1-0
• mqtt
• +4 more

65.numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

• kubernetes
• stream-processing
• data-processing
• k8s
• pipeline
• map-reduce
• +1 more

66.feast-dev / feast

The Open Source Feature Store for AI/ML

• machine-learning
• features
• ml
• big-data
• feature-store
• python
• +4 more

67.aeron-io / aeron

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

• messaging
• java
• c-plus-plus
• ipc
• multicast-streams
• c

68.macrozheng / mall-swarm

mall-swarm是一套微服务商城系统，采用了 Spring Cloud Alibaba、Spring Boot 3.5、Sa-Token、MyBatis、Elasticsearch、Docker、Kubernetes等核心技术，同时提供了基于Vue的管理后台方便快速搭建系统。mall-swarm在电商业务的基础集成了注册中心、配置中心、监控中心、网关等系统功能。

• springcloud
• springboot
• docker
• java
• mysql
• elasticsearch
• +8 more

69.vesoft-inc / nebula

A distributed, fast open-source graph database featuring horizontal scalability and high availability

• graph-database
• distributed
• database
• graphdb
• raft
• cpp
• +11 more

70.apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler
• job-scheduler
• cloud-native
• +5 more

71.online-ml / river

🌊 Online machine learning in Python

• incremental-learning
• machine-learning
• python
• online-learning
• online-statistics
• data-science
• +6 more

72.apache / rocketmq

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.

• rocketmq
• cloud-native
• eventing
• java
• messaging
• streaming
• +1 more

73.tinode / chat

Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots

• chat
• chat-application
• chatapp
• go
• golang
• instant-messaging
• +14 more

74.provectus / kafka-ui

Open-Source Web UI for Apache Kafka Management

• kafka-ui
• kafka-brokers
• kafka
• kafka-streams
• kafka-client
• opensource
• +12 more

75.delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics
• delta-lake

76.catboost / catboost

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

• machine-learning
• decision-trees
• gradient-boosting
• gbm
• gbdt
• python
• +12 more

77.clidey / whodb

Where data access meets operational intelligence

• mysql
• postgresql
• ai-agent
• data-catalog
• data-governance
• data-integration
• +14 more

78.ThreeDotsLabs / watermill

Building event-driven applications the easy way in Go.

• watermill
• event-driven
• event-sourcing
• sagas
• events
• stream-processing
• +7 more

79.svix / svix-webhooks

The open source and enterprise-ready webhooks service 🦀

• webhooks
• rust
• webhook-server
• webhook-service
• webhook
• webhooks-server
• +13 more

80.ag2ai / faststream

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

• asyncio
• kafka
• python
• rabbitmq
• asyncapi
• distributed-systems
• +7 more

81.uNetworking / uWebSockets

Simple, secure & standards compliant web server for the most demanding of applications

• websockets
• http
• pubsub
• router
• commercial
• cross-platform
• +1 more

82.apache / hadoop

Apache Hadoop

• hadoop

83.cython / cython

The most widely used Python to C compiler

• python
• cython
• cpython
• cpython-extensions
• c
• cpp
• +2 more

84.nats-io / nats.go

Golang client for NATS, the cloud native messaging system.

• go
• golang
• nats
• microservices
• microservices-architecture
• pub-sub
• +3 more

85.heibaiying / BigData-Notes

大数据入门指南 :star:

• hadoop
• hdfs
• yarn
• mapreduce
• hive
• spark
• +11 more

86.heynickc / awesome-ddd

A curated list of Domain-Driven Design (DDD), Command Query Responsibility Segregation (CQRS), Event Sourcing, and Event Storming resources

• awesome
• awesome-list
• list

87.FavioVazquez / ds-cheatsheets

List of Data Science Cheatsheets to rule the world

• datascience
• python
• r
• spark
• programming
• jupyter
• +1 more

88.treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data

• data-engineering
• data-versioning
• go
• object-storage
• data-lake
• aws-s3
• +13 more

89.PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

• etl
• sql
• bigquery
• cloud-native
• distributed-systems
• kafka
• +10 more

90.apache / cloudberry

One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.

• database
• data-analysis
• data-warehouse
• olap
• postgresql
• sql
• +8 more

91.bruin-data / bruin

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines
• python
• snowflake
• +5 more

92.alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件

93.arkime / arkime

Arkime is an open source, large scale, full packet capturing, indexing, and database system.

• network-monitoring
• pcap
• packet-capture
• nsm
• javascript
• c
• +2 more

94.apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.

• hudi
• apachehudi
• datalake
• bigdata
• apachespark
• incremental-processing
• +3 more

95.zeromq / libzmq

ZeroMQ core engine in C++, implements ZMTP/3.1

• zeromq
• zmq
• messaging
• concurrency
• pubsub
• pushpull
• +5 more

96.yudaocode / SpringBoot-Labs

一个涵盖六个专栏：Spring Boot 2.X、Spring Cloud、Spring Cloud Alibaba、Dubbo、分布式消息队列、分布式事务的仓库。希望胖友小手一抖，右上角来个 Star，感恩 1024

• spring-boot
• spring-cloud
• spring-cloud-alibaba
• dubbo
• rocketmq
• rabbitmq
• +2 more

97.apache / camel

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices
• cloud-native
• data-transformation
• +10 more

98.turbot / steampipe

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

• steampipe
• postgresql
• postgresql-fdw
• cloud
• security
• aws
• +14 more

99.apache / pulsar

Apache Pulsar - distributed pub-sub messaging system

• pulsar
• pubsub
• messaging
• streaming
• queuing
• event-streaming

100.ThreeDotsLabs / wild-workouts-go-ddd-example

Go DDD example application. Complete project to show how to apply DDD, Clean Architecture, and CQRS by practical refactoring.

• go
• golang
• ddd
• clean-architecture
• hexagonal-architecture
• cqrs
• +7 more