Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Repository Index
100,822 visible repositories in the index
1–15 of 15 · Most stars
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
- data
- data-analysis
- data-engineering
- data-integration
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。
- mongo-database
- mongodb-oplog
- replication
- active-active
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
- elasticsearch
- go
- golang
- mongodb
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
- logstash
- elk
- kafka
- elasticsearch
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
- transform
- etl
- xml
- json
Distributed Data Transfer Service for MySQL
- mysql
- data-transfer
- data-migration
- replication
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
The data platform for manufacturing
- oee
- iot
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
go-etl is a toolset for data extraction, transformation and loading.
- etl
- mysql
- go
- postgresql
Track Go benchmark performance over time by storing results in InfluxDB