Django application and library for importing and exporting data with admin integration.
Repository Index
95,842 visible repositories in the index
25–48 of 86 · Most stars
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Spring Batch is a framework for writing batch applications using Java and Spring
- batch
- batch-processing
- java
- spring
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
- claude-code
- codex
- data-processing
- harness-engineering
深圳地铁大数据客流分析系统🚇🚄🌟
Flow collector, enricher and visualizer
- collector
- netflow
- sflow
- ipfix
A high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary.
- logical-replication
- postgres
- replication
- rust
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
- unstructured-data
- low-code
- developer-tools
- etl
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- sql
- redpanda
- single-binary
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
- data
- ingestion
- replication
- management
Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.
Python CLI utility and library for manipulating SQLite databases
- sqlite
- python
- datasette
- sqlite-database
Bifrost ---- 面向生产环境的 MySQL,MariaDB,kafka 同步到Redis,MongoDB,ClickHouse,StarRocks,Doris,Kafka等服务的异构中间件
- mysqlbinlog
- mysql-redis
- mysql-mongodb
- go
superglue's AI agents connect, migrate and implement enterprise systems. Cloud-hosted or on your own infrastructure.
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
Scalable data pre processing and curation toolkit for LLMs
- data-curation
- llm
- data
- data-prep
MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。
- mongo-database
- mongodb-oplog
- replication
- active-active
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
The ToroDB solution to provide better analytics on top of MongoDB and make it easier to migrate from MongoDB to SQL
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Python Streaming DataFrames for Kafka
Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.
- apis
- data
- rust
- sql