For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.
Repository Index
55,140 visible repositories in the index
1–24 of 270 · Most stars
Open Data Platform for analysts, quants and AI agents.
- python
- machine-learning
- finance
- stocks
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
Apache Flink
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
- python
- workflow
- data-engineering
- data-science
All-in-one IoT Platform - Device management, data collection, processing and visualization.
- visualization
- iot
- iot-platform
- mqtt
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
GoDS (Go Data Structures) - Sets, Lists, Stacks, Maps, Trees, Queues, and much more
- go
- golang
- data-structure
- map
TranslatedDataX is the open-source version of Alibaba Cloud DataWorks Data Integration.
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
- metadata
- datadiscovery
- dataquality
- data-profiling
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Logstash - transport and process your logs, events, or other data
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
- workflow-schedule
- azkaban
- airflow
- task-scheduler
Graph-Native Infrastructure for Context and Accountable AI Systems
- ai
- ai-governance
- artificial-intelligence
- context-engineering
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
Parallel computing with task scheduling
- dask
- python
- pydata
- numpy
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
- pandas-profiling
- pandas-dataframe
- statistics
- jupyter-notebook