Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
77,852 visible repositories in the index
1–24 of 46 · Most stars
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Migrate to PostgreSQL in a single command!
- postgresql
- migration
- mysql
- mssql
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Flink CDC is a streaming data integration tool
- change-data-capture
- cdc
- batch
- data-integration
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
Sync MySQL data into elasticsearch
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
Sync data from one Postgres database to another
Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
- garmin
- python
- health
- database
Scalable and efficient data transformation framework - backwards compatible with dbt.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Database Subsetting and Relational Data Browsing Tool.
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
sq data wrangler
阿里巴巴去Oracle数据迁移同步工具(全量+增量,目标支持MySQL/DRDS)
深圳地铁大数据客流分析系统🚇🚄🌟
A high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary.
- logical-replication
- postgres
- replication
- rust
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- sql
- redpanda
- single-binary
Python CLI utility and library for manipulating SQLite databases
- sqlite
- python
- datasette
- sqlite-database
Bifrost ---- 面向生产环境的 MySQL,MariaDB,kafka 同步到Redis,MongoDB,ClickHouse,StarRocks,Doris,Kafka等服务的异构中间件
- mysqlbinlog
- mysql-redis
- mysql-mongodb
- go
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql