Repository Index

Fancy stream processing made operationally mundane

• message-queue
• stream-processing
• streaming-data
• message-bus

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

• aws
• gcp
• azure
• sql

Privacy and Security focused Segment-alternative, in Golang and React

• privacy
• warehouse-management
• data-warehouse
• customer-data

RedisShake is a Valkey/Redis data processing and migration tool.

• redis
• redis-cluster
• redis-proxy

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

• bigquery
• copy-database
• data-ingestion
• data-integration

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

• data
• data-analysis
• data-engineering
• data-integration

Connect your company's data, let an agent draft an ontology over it, and load it into HelixDB as one graph you can explore and ask questions of. Hosted at foundry.helix-db.com, or run it on your own machine.

• ai-agents
• data-integration
• golang
• graph-database

MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具，可以满足迁移和同步的需求，进一步实现灾备和多活功能。

• mongo-database
• mongodb-oplog
• replication
• active-active

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

• go
• data-integration
• etl
• data-pipeline

Distributed Data Transfer Service for MySQL

• mysql
• data-transfer
• data-migration
• replication

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)

• cdc
• tidb
• mysql
• kafka

A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.

go-etl is a toolset for data extraction, transformation and loading.

• etl
• mysql
• go
• postgresql