Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
- go
- data-integration
- etl
- data-pipeline
Repository Index
47,133 repositories indexed
49–72 of 86 · Most stars
Engine for AI/ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
- pandas
- pandas-summary
- dataframes
- data-science
Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.
- data-analytics
- data-cleaning
- data-cleansing
- data-engineering
Simple but powerful DAG scheduler and dashboard
- directed-acyclic-graph
- airflow
- dashboard
- go
Free and open source schema versioning and database migration made natively with .NET/6. NEW THIS MAY 2022! v1.3.15 released!
LittleHorse Server: Business-as-Code Platform for Microservices, Workflows, and Agents
High-performance, time-indexed object storage for robotics and industrial IoT
- blob-storage
- storage-engine
- storage
- historian
The repository host gives no description.
- hacktoberfest
- data
- datalineage
- dbt
The data transformation tool the open-source community deserves.
Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
- apache-iceberg
- apache-spark
- data-engineering
- delta-lake
The type-safe, event-driven workflow orchestration library that scales with your business. Build robust, distributed workflows in Go with compile-time safety, automatic retries, and horizontal scaling out of the box. Integrate with your existing tech stack using adapters 🚀
- eventdrivenarchitecture
- golang
- kafka
- reflex
Lightweight stream processing engine for IoT
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.
- python3
- data-science
- machine-learning
- python
Machine is a workflow/pipeline library for processing data
- golang
- golang-library
- golang-package
- workflow
Data-mining and Visualization components for RAD Studio
- visualization
- complex-structures
- database
- charting-library
Time-series database and pub/sub messaging server for IoT and real-time devices, in Go: an embeddable storage engine with TTLs, wildcard topics and encryption, and a clustered server over gRPC, TCP and WebSocket with replication and rolling upgrades.
Distributed batch data processing framework
Multi-core Window-Based Stream Processing Engine
- llvm
- incremental-computation
- aggregation
- stream-processing
Realistic synthetic events for testing, demos, and pipelines — streamed live or generated in bulk
- clickhouse
- data-engineering
- data-generation
- devops
Lightweight SQL-based stream processing engine for IoT edge.
A lightweight Snowflake emulator built with Go and DuckDB for local development and testing
Open-source streaming SQL engine written in Rust using Apache Arrow and DataFusion. Supports continuous queries, temporal stream joins, tumbling/session windows, and CDC/Kafka connectors. Lightweight, embeddable, and sub-microsecond latency
Stream Ops is a fully embeddable data streaming engine and stream processing API for Java.
Window-Based Hybrid CPU/GPU Stream Processing Engine
- stream-processing
- hybrid
- multicore
- gpu