Repository Index

Database Subsetting and Relational Data Browsing Tool.

• database
• subsetting
• frontend
• export

Self-hosted platform to unify wearable health data through one AI-ready API.

• wearables
• apple-health
• apple-watch
• fitness

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

• dataops
• dataops-platform
• elt
• open-source

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

• data
• ingestion
• replication
• management

Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.

• dataprep
• data-science
• datapreparation
• dataconnector

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.

• iot
• iot-gateway
• gateway
• mqtt

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

• etl
• data-integration
• python
• postgresql

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

Apache Spark to Apache Cassandra connector

• cassandra
• scala
• spark

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines

Embulk: Pluggable Bulk Data Loader.

• embulk
• bulk-loader

Data processing & ETL framework for Ruby

• etl
• etl-ruby
• ruby
• data

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

• data-engineering
• reverse-etl
• data-pipeline
• data-activation

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering

Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

• data-engineering
• data-integration
• data-pipeline
• data-quality

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

Apache InLong - a one-stop, full-scenario integration framework for massive data

• inlong
• one-stop-service
• data-streaming
• event-streaming

visual data prep powered by python

• data
• data-pipelines
• etl
• structured-data

Import CSV and JSON into PostgreSQL the easy way

Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.

• data-science
• data-visualization
• visualization
• data-analysis

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream