Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–24 of 45 · Most stars
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
LFX Community Data Platform (CDP)
- community
- cdp
- open-source
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
- transform
- etl
- xml
- json
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
- change-data-capture
- data
- data-collection
- data-engineering
Healthcare data infrastructure for next generation care delivery.
- api
- healthcare
- open-source
- soc2
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
Amazon Redshift Database Loader implemented in AWS Lambda
🌱 The ultimate solution for populating your MongoDB database.
Distributed Data Transfer Service for MySQL
- mysql
- data-transfer
- data-migration
- replication
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Graph-based security analysis for everyone
Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰
- csv-import
- data-import
- data-mapping
- data-review
Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.
A plugin to push to Algolia based on graphQl queries
- gatsby
- algolia
- search
- static-site-generator