Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–24 of 43 · Most stars
TranslatedDataX is the open-source version of Alibaba Cloud DataWorks Data Integration.
Logstash - transport and process your logs, events, or other data
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Pentaho Data Integration ( ETL ) a.k.a Kettle
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
Apache NiFi
- nifi
- java
- hacktoberfest
- apache
Database Subsetting and Relational Data Browsing Tool.
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
- data
- ingestion
- replication
- management
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
Embulk: Pluggable Bulk Data Loader.
- embulk
- bulk-loader
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.
The swiss army knife of healthcare integration.
- mirth
- mirth-connect
- hl7
- healthcare
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
- transform
- etl
- xml
- json
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
The MongoDB Spark Connector
The simple, stupid batch framework for Java
- java
- batch-processing
- batch
Amazon Redshift Database Loader implemented in AWS Lambda
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
Apache HBase Connectors
- database
- java
- hbase
Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.
AtlasMap project repository
- atlasmap
A plugin to push to Algolia based on graphQl queries
- gatsby
- algolia
- search
- static-site-generator