Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
- python
- workflow
- data-engineering
- data-science
100,822 visible repositories in the index
1–24 of 24 · Most stars
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
An orchestration platform for the development, production, and observation of data assets.
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
Reactive Extensions for the Go language.
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Database Subsetting and Relational Data Browsing Tool.
REST job server for Apache Spark
Data Contracts engine for the modern data stack. https://www.soda.io
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
👜 Tiny and fast reactive/iterable programming library
re_data - fix data issues before your users & CEO would discover them 😊
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
Zero-cost & Type-safe Reactive Extensions for Rust. Supports both high-performance single-threaded (Local) and thread-safe (Shared) contexts. WASM & Tokio ready.
📙 Awesome Data Catalogs and Observability Platforms.
🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications
An observable microservice bus library for .NET, that wraps the underlying message transports in simple Rx based interfaces.
Data Quality and Observability platform for the whole data lifecycle, from profiling new data sources to full automation with Data Observability. Configure data quality checks from the UI or in YAML files, let DQOps run the data quality checks daily to detect data quality issues.
Reactive queryable observable framework.
Scalable Full History API Solution for Antelope (former EOSIO) based blockchains
Declarative data quality engine. Define checks in YAML, run anywhere.
A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.