Repository Index

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code

Apache Flink

• scala
• java
• big-data
• flink

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science

All-in-one IoT Platform - Device management, data collection, processing and visualization.

• visualization
• iot
• iot-platform
• mqtt

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql

Turns Data and AI algorithms into production-ready web applications in no time.

• automation
• data-engineering
• data-ops
• data-visualization

An open source cybersecurity protocol for syncing decentralized graph data.

• machine-learning
• artificial-intelligence
• big-data
• blockchain

🏆 Real-Time no-code, powerful and secure ORM 🚀 providing APIs and Docs without coding by Backend, and Frontend(Client) can customize response JSONs 🏆 实时 零代码、全功能、强安全 ORM 库 🚀 后端接口和文档零代码，前端(客户端) 定制返回 JSON 的数据和结构

• postgresql-database
• tidb
• sqlserver
• oracle

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

• telegraf
• monitoring
• time-series
• metrics

QuestDB is a high performance, open-source, time-series database

• time-series
• low-latency
• database
• sql

GoDS (Go Data Structures) - Sets, Lists, Stacks, Maps, Trees, Queues, and much more

• go
• golang
• data-structure
• map

Workflow Engine for Kubernetes

• workflow
• kubernetes
• argo
• dag

大数据入门指南 :star:

• hadoop
• hdfs
• yarn
• mapreduce

The official home of the Presto distributed SQL query engine for big data

• java
• presto
• hive
• hadoop

List of Data Science Cheatsheets to rule the world

• datascience
• python
• r
• spark

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science

Apache Doris is a real-time analytics and hybrid search database for AI agents.

• olap
• database
• hudi
• iceberg

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp

The Data Engineering Cookbook

• data-engineer
• data-engineering
• big-data
• best-practices