高性能表格数据导出器
Repository Index
105,102 visible repositories in the index
73–96 of 171 · Most stars
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。
- mongo-database
- mongodb-oplog
- replication
- active-active
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
The ToroDB solution to provide better analytics on top of MongoDB and make it easier to migrate from MongoDB to SQL
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Database Reporting Tool and Tasks (.Net)
- reporting-engine
- report-generator
- business-intelligence
- task-scheduler
Extract Transform Load for Python 3.5+
- data-processing
- extract-transform-load
- bonobo
- python3
cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes
Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)
Python Streaming DataFrames for Kafka
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 427 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
- etl-pipeline
- etl-framework
- spark
- apache-spark
Hop Orchestration Platform
- java
- streaming
- hop
- apache
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Concurrent Python made simple
- asyncio
- concurrency
- python
- threading
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
Elasticsearch File System Crawler (FS Crawler)
- java
- elasticsearch
- crawler
- tika
Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.
- hadoop
- hive
- database
- clickhouse
Support agile Ontology DataOps Based on Flink, DataX and Flink-CDC with Web-UI
Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync
- elasticsearch
- elasticsearch-sync
- postgresql
- change-data-capture
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.
- blockchain
- blockchain-indexer
- ethereum
- etl
Import CSV and JSON into PostgreSQL the easy way