MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。
- mongo-database
- mongodb-oplog
- replication
- active-active
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
Extract Transform Load for Python 3.5+
- data-processing
- extract-transform-load
- bonobo
- python3
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 446 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Hop Orchestration Platform
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
Python Extract Transform and Load Tables of Data
Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
- apache-beam
- dataflow-templates
- google-cloud-dataflow
- google-cloud-storage
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
The swiss army knife of healthcare integration.
- mirth
- mirth-connect
- hl7
- healthcare
Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.
Scratch is a swiss army knife for big data.
Build a data warehouse in SQL Server from two source systems, a CRM and an ERP, through bronze, silver and gold. The files and scripts of the free full project.
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
MetaForge is integrated software for data transformation, integration, and visualization. Support a variety of data sources, file types, big data related database, relational database, NoSQL database, etc. Through the software can realize the management of multiple data sources, the data under the source of various operations conversion, making dat
Dataform is a framework for managing SQL based data operations in BigQuery
Input pipeline framework
qData is a lightweight, feature-complete, AI-native open-source data platform. It covers data integration, development, modeling, governance, assets and API services, with self-hosted deployment, flexible scaling and commercial use.
- data-middle-platform
- data-development
- data-integration
- data-modeling
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)