Repository Index

TranslatedDataX is the open-source version of Alibaba Cloud DataWorks Data Integration.

Logstash - transport and process your logs, events, or other data

• etl-framework
• streaming
• logging
• java

OpenRefine is a free, open source power tool for working with messy data and improving it

• datacleansing
• data-analysis
• java
• opendata

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time

Pentaho Data Integration ( ETL ) a.k.a Kettle

Enso Analytics is a self-service data prep and analysis platform designed for data teams.

• visual
• language
• functional
• compiler

Flink CDC is a streaming data integration tool

• change-data-capture
• cdc
• batch
• data-integration

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices

Apache NiFi

• nifi
• java
• hacktoberfest
• apache

TranslatedDataX integrated with a visual page: select a data source and generate a data synchronization task with one click. Supports data sources such as RDBMS, Hive, HBase, ClickHouse, and MongoDB, and batch creation of RDBMS data synchronization tasks. Integrates an open-source scheduling system, and supports distributed and incremental data synchronization, real-time viewing of running logs, monitoring of executor resources, KILL of running processes, encryption of data source information, and more.

Database Subsetting and Relational Data Browsing Tool.

• database
• subsetting
• frontend
• export

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

• data
• ingestion
• replication
• management

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

Embulk: Pluggable Bulk Data Loader.

• embulk
• bulk-loader

Apache InLong - a one-stop, full-scenario integration framework for massive data

• inlong
• one-stop-service
• data-streaming
• event-streaming

The swiss army knife of healthcare integration.

• mirth
• mirth-connect
• hl7
• healthcare

📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)

• analytics
• bi-server
• java
• big-data

The MongoDB Spark Connector

• spark
• spark-packages
• mongo-spark
• mongodb

The simple, stupid batch framework for Java

• java
• batch-processing
• batch

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

• java
• smooks
• enterprise-integration
• etl

AtlasMap project repository

• atlasmap

RDF-Centric Map/Reduce Framework and Freebase data conversion tool

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.