Trending repositories

1.DataExpert-io / data-engineer-handbook

This is a repo with links to everything you'd ever want to learn about data engineering

• apachespark
• awesome
• bigdata
• data
• dataengineering
• sql

2.hugues31 / duckpad

The data transformation tool the open-source community deserves.

• data-engineering
• duckdb

3.rustfs / rustfs

RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

• bigdata
• cloud-native
• filesystem
• object-storage
• rust
• s3
• +6 more

4.permissionlesstech / bitchat

bluetooth mesh chat, IRC vibes

• bluetooth
• bluetooth-le
• decentralized
• e2e-encryption
• ios
• macos
• +4 more

5.binhnguyennus / awesome-scalability

The Patterns of Scalable, Reliable, and Performant Large-Scale Systems

• system-design
• backend
• scalability
• interview
• architecture
• devops
• +14 more

6.DataTalksClub / data-engineering-zoomcamp

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

• data-engineering
• kafka
• spark
• dbt
• docker
• kestra
• +2 more

7.binwiederhier / ntfy

Send push notifications to your phone or desktop using PUT/POST

• rest-api
• pubsub
• notifications
• curl
• push-notifications
• ntfy
• +1 more

8.stepll2026 / OntiCards

OntiCards 是面向企业的 AI 数据中枢（AI Data Hub），围绕业务数据库提供一站式能力：数据源连接、元数据理解、自然语言查询、跨源融合、数据质检、查询审计与数据资产管理等

9.ModernRelay / omnigraph

Object-storage native graph database with branching and typed ontology

• apache-arrow
• datafusion
• graph-database
• knowledge-graph
• lakehouse
• lance
• +5 more

10.ClickHouse / ClickHouse

ClickHouse® is a real-time analytics database management system

• dbms
• olap
• analytics
• sql
• big-data
• mpp
• +11 more

11.kestra-io / kestra

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code
• automation
• devops
• +14 more

12.google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

• mediapipe
• c-plus-plus
• computer-vision
• deep-learning
• android
• video-processing
• +11 more

13.mlflow / mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

• machine-learning
• ai
• ml
• mlflow
• apache-spark
• model-management
• +12 more

14.apache / superset

Apache Superset is a Data Visualization and Data Exploration Platform

• superset
• apache
• apache-superset
• data-visualization
• data-viz
• analytics
• +12 more

15.dagucloud / dagu

Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.

• cron
• workflow-engine
• workflow-scheduler
• workflow-orchestration
• devops
• data-pipeline
• +14 more

16.marimo-team / marimo

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.

• notebooks
• python
• data-science
• machine-learning
• artificial-intelligence
• data-visualization
• +7 more

17.apache / spark

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java
• big-data
• jdbc
• +2 more

18.GokuMohandas / Made-With-ML

Learn how to develop, deploy and iterate on production-grade ML applications.

• machine-learning
• deep-learning
• pytorch
• natural-language-processing
• data-science
• python
• +8 more

19.apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python
• scheduler
• workflow
• +14 more

20.nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.

• go
• golang
• messaging
• message-bus
• message-queue
• cloud-native
• +9 more

21.cocoindex-io / cocoindex

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python
• +14 more

22.eugeneyan / applied-ml

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

• applied-machine-learning
• production
• applied-data-science
• machine-learning
• data-science
• reinforcement-learning
• +8 more

23.macrozheng / mall

mall项目是一套电商系统，包括前台商城系统及后台管理系统，基于Spring Boot+MyBatis实现，采用Docker容器化部署。 前台商城系统包含首页门户、商品推荐、商品搜索、商品展示、购物车、订单流程、会员中心、客户服务、帮助中心等模块。 后台管理系统包含商品管理、订单管理、会员管理、促销管理、运营管理、内容管理、统计报表、财务管理、权限管理、设置等模块。

• spring-boot
• spring-security
• mybatis
• docker
• swagger-ui
• mysql
• +10 more

24.argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes

• argo
• kubernetes
• continuous-deployment
• gitops
• continuous-delivery
• docker
• +10 more

25.dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science
• workflow-automation
• python
• +9 more

26.Unstructured-IO / unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp
• ocr
• information-retrieval
• +14 more

27.riverqueue / river

The polyglot queue: Fast and reliable background jobs in Go, Java, JS, Ruby, and Rust on Postgres/SQLite.

• background-jobs
• go
• golang
• postgres
• postgresql
• queue
• +5 more

28.BlueRingsLabs / no-shortcuts

Free Course : Data, Machine Learning, Deep Learning and AI Engineering, the long way round: the math, the code, the systems and the security. 173 lessons, 166 runnable labs, no shortcuts. Free under CC BY-NC 4.0.

• ai-engineering
• course
• data-engineering
• deep-learning
• llm
• machine-learning
• +6 more

29.restatedev / restate

Restate is the platform for building resilient applications that tolerate all infrastructure faults w/o the need for a PhD.

• distributed-systems
• async-await
• durable-execution
• microservices
• consistent-state
• reliable-communication
• +4 more

30.debezium / debezium

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium
• cdc
• database
• +5 more

31.dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake
• data-loading
• data-warehouse
• +4 more

32.PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science
• workflow-engine
• prefect
• +8 more

33.airbytehq / airbyte

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering
• java
• python
• +14 more

34.projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.

• http
• bugbounty
• ssl-certificate
• pipeline
• pentest-tool
• cybersecurity
• +4 more

35.hatchet-dev / hatchet

🪓 An orchestration engine for background tasks, AI agents, and durable workflows

• event-driven
• golang
• workflow-engine
• distributed
• distributed-systems
• fastapi
• +7 more

36.timgit / pg-boss

Queueing jobs in Postgres from Node.js like a boss

• postgresql
• postgres
• queue
• nodejs
• node
• job

37.simplex-chat / simplex-chat

SimpleX - the first messaging network operating without user identifiers of any kind - 100% private by design! iOS, Android and desktop apps 📱!

• chat
• protocol
• e2ee
• privacy
• haskell
• security
• +3 more

38.dbt-labs / dbt

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

• dbt-viewpoint
• slack
• pypa
• data-modeling
• business-intelligence
• analytics
• +1 more

39.thingsboard / thingsboard

All-in-one IoT Platform - Device management, data collection, processing and visualization.

• visualization
• iot
• iot-platform
• mqtt
• middleware
• thingsboard
• +12 more

40.apache / iggy

Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed

• messaging
• quic
• rust
• streaming
• http
• iggy
• +2 more

41.redis / redis

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

• database
• key-value
• nosql
• redis
• caching
• distributed-systems
• +14 more

42.vectordotdev / vector

A high-performance observability data pipeline.

• logs
• metrics
• observability
• forwarder
• events
• stream-processing
• +12 more

43.argoproj / argo-workflows

Workflow Engine for Kubernetes

• workflow
• kubernetes
• argo
• dag
• knative
• airflow
• +12 more

44.trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

• java
• presto
• hive
• hadoop
• big-data
• sql
• +13 more

45.eclipse-mosquitto / mosquitto

Eclipse Mosquitto - An open source MQTT broker

• mqtt
• broker
• mosquitto
• eclipse-iot
• iot
• messaging
• +3 more

46.growthbook / growthbook

Open Source Feature Flags, Experimentation, and Product Analytics

• abtesting
• statistics
• abtest
• experimentation
• split-testing
• snowflake
• +14 more

47.juicedata / juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

• filesystem
• cloud-native
• golang
• redis
• distributed-systems
• storage
• +6 more

48.taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL

• background-jobs
• nodejs
• queue
• redis
• typescript
• python
• +3 more

49.apache / doris

Apache Doris is a real-time analytics and hybrid search database for AI agents.

• olap
• database
• hudi
• iceberg
• real-time
• sql
• +12 more

50.tobymao / sqlglot

Python SQL Parser and Transpiler

• transpiler
• sql
• python
• parser
• optimizer
• bigquery
• +14 more

51.inngest / inngest

The leading workflow orchestration platform. Run stateful step functions and AI workflows on serverless, servers, or the edge.

• cli
• event-driven
• serverless-functions
• queues
• serverless
• event-driven-architecture
• +2 more

52.apache / seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time
• apache
• batch
• +8 more

53.dragonflydb / dragonfly

A modern replacement for Redis and Memcached

• memcached
• multi-threading
• redis
• cpp
• fibers
• cache
• +9 more

54.questdb / questdb

QuestDB is a high performance, open-source, time-series database

• time-series
• low-latency
• database
• sql
• questdb
• tsdb
• +14 more

55.centrifugal / centrifugo

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

• websocket
• redis
• real-time
• messaging
• scalability
• eventsource
• +14 more

56.kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

• kubernetes
• serverless
• autoscaling
• event-driven
• keda
• hacktoberfest

57.lightdash / lightdash

Agentic BI. Analytics at the speed of code ⚡️

• dbt
• data-visualization
• data-analytics
• business-intelligence

58.StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

• database
• olap
• sql
• analytics
• big-data
• realtime-database
• +14 more

59.evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning
• model-monitoring
• html-report
• +8 more

60.evidence-dev / evidence

Business intelligence as code: build fast, interactive data visualizations in SQL and markdown

• analytics
• sql
• business-intelligence
• data-visualization
• dbt
• duckdb
• +8 more

61.dapr / dapr

Dapr is a portable runtime for building distributed applications across cloud and edge, combining event-driven architecture with workflow orchestration.

• microservices
• microservice
• kubernetes
• sidecar
• state-management
• event-driven
• +8 more

62.oxnr / awesome-bigdata

A curated list of awesome big data frameworks, ressources and other awesomeness.

• awesome
• bigdata
• data-warehouse
• data-analytics
• data-visualization
• data-stream
• +9 more

63.openimsdk / open-im-server

IM Chat OpenClaw

• im
• chat
• messaging
• messenger
• openclaw

64.apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler
• job-scheduler
• cloud-native
• +5 more

65.kgrzybek / modular-monolith-with-ddd

Full Modular Monolith application with Domain-Driven Design approach.

• ddd
• ddd-architecture
• cqrs
• architecture
• rest-api
• dotnet
• +14 more

66.apache / flink

Apache Flink

• scala
• java
• big-data
• flink
• python
• sql

67.quickwit-oss / quickwit

Cloud-native OSS search engine for observability

• rust
• log-management
• logs
• tantivy
• cloud-native
• open-source
• +4 more

68.apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query
• stratosphere
• metalake
• +5 more

69.pgmq / pgmq

A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.

• message-queue

70.influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

• telegraf
• monitoring
• time-series
• metrics
• golang
• influxdb
• +10 more

71.emqx / emqx

The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles

• mqtt
• iot
• mqtt-broker
• erlang
• iot-middleware
• broker
• +14 more

72.macrozheng / mall-swarm

mall-swarm是一套微服务商城系统，采用了 Spring Cloud Alibaba、Spring Boot 3.5、Sa-Token、MyBatis、Elasticsearch、Docker、Kubernetes等核心技术，同时提供了基于Vue的管理后台方便快速搭建系统。mall-swarm在电商业务的基础集成了注册中心、配置中心、监控中心、网关等系统功能。

• springcloud
• springboot
• docker
• java
• mysql
• elasticsearch
• +8 more

73.bytewax / bytewax

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering
• data-processing
• data-science
• +3 more

74.yeasy / docker_practice

最新Docker容器技术，从真实案例中学习最佳实践！| Learn and understand Docker&Container technologies, with real DevOps practice!

• docker
• book
• cloud-computing
• container
• kubernetes
• swarm
• +5 more

75.taosdata / TDengine

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

• iot
• bigdata
• time-series
• database
• industrial-iot
• connected-vehicles
• +11 more

76.vesoft-inc / nebula

A distributed, fast open-source graph database featuring horizontal scalability and high availability

• graph-database
• distributed
• database
• graphdb
• raft
• cpp
• +11 more

77.rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins

• rabbitmq
• amqp
• messaging
• amqp-0-9-1
• amqp1-0
• mqtt
• +4 more

78.zeromq / libzmq

ZeroMQ core engine in C++, implements ZMTP/3.1

• zeromq
• zmq
• messaging
• concurrency
• pubsub
• pushpull
• +5 more

79.catboost / catboost

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

• machine-learning
• decision-trees
• gradient-boosting
• gbm
• gbdt
• python
• +12 more

80.apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam
• batch
• golang
• +2 more

81.apache / camel

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices
• cloud-native
• data-transformation
• +10 more

82.confluentinc / librdkafka

The Apache Kafka C/C++ library

• kafka
• kafka-consumer
• apache-kafka
• high-performance
• c
• librdkafka
• +3 more

83.apache / hadoop

Apache Hadoop

• hadoop

84.tinode / chat

Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots

• chat
• chat-application
• chatapp
• go
• golang
• instant-messaging
• +14 more

85.fivetran / great_expectations

Always know what to expect from your data.

• pipeline-tests
• dataquality
• datacleaning
• datacleaner
• data-science
• data-profiling
• +14 more

86.bruin-data / ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

• bigquery
• copy-database
• data-ingestion
• data-integration
• data-pipeline
• duckdb
• +4 more

87.slothflowlabs / duckle

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

• data-engineering
• data-integration
• data-pipeline
• data-quality
• duckdb
• elt
• +14 more

88.redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

• kafka
• cpp
• kubernetes
• go
• streaming
• redpanda
• +7 more

89.madd86 / awesome-system-design

A curated list of awesome System Design (A.K.A. Distributed Systems) resources.

• relational-database
• message-broker
• hadoop-ecosystem
• nosql
• interview
• distributed-systems
• +3 more

90.risingwavelabs / risingwave

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql
• kafka
• materialized-view
• +5 more

91.delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics
• delta-lake

92.apecloud / kubeblocks

KubeBlocks is a Kubernetes Operator designed to manage a variety of databases and streaming systems, including MySQL, PostgreSQL, MongoDB, Redis, RabbitMQ, RocketMQ, and more, within Kubernetes environments.

• mysql-operator
• mongodb-operator
• postgresql-operator
• rabbitmq-operator
• redis-operator
• databases
• +13 more

93.cython / cython

The most widely used Python to C compiler

• python
• cython
• cpython
• cpython-extensions
• c
• cpp
• +2 more

94.ibis-project / ibis

the portable Python dataframe library

• python
• impala
• pandas
• database
• clickhouse
• postgresql
• +12 more

95.svix / svix-webhooks

The open source and enterprise-ready webhooks service 🦀

• webhooks
• rust
• webhook-server
• webhook-service
• webhook
• webhooks-server
• +13 more

96.bruin-data / bruin

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines
• python
• snowflake
• +5 more

97.containrrr / shoutrrr

Notification library for gophers and their furry friends.

• notifications
• golang
• go
• messaging
• integrations
• hacktoberfest

98.amark / gun

An open source cybersecurity protocol for syncing decentralized graph data.

• machine-learning
• artificial-intelligence
• big-data
• blockchain
• p2p
• decentralized
• +14 more

99.PipedreamHQ / pipedream

Connect APIs, remarkably fast. Free for developers.

• eventsourcing
• event-sourcing
• automation
• workflow
• workflows
• apis
• +13 more

100.apache / couchdb

Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability

• content
• network-server
• http
• cloud
• erlang
• javascript
• +4 more