Repository Index

Distributed Data Transfer Service for MySQL

• mysql
• data-transfer
• data-migration
• replication

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)

• cdc
• tidb
• mysql
• kafka

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

• java
• smooks
• enterprise-integration
• etl

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs

• spark
• neo4j-connector
• cypher
• neo4j-driver

Apache HBase Connectors

• database
• java
• hbase

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

Event-driven OneLake pipeline with automated Upsert merge and Python reporting

• activator
• data-engineering
• data-factory
• data-pipeline

Scalable Full History API Solution for Antelope (former EOSIO) based blockchains

• antelope
• api
• big-data
• blockchain

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

A high performance data transfer tool that can be used to transfer data from Bitcoin Core blockchain files to a SQL Server database

• bitcoin

This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.

• magento
• import
• products
• categories

🇪🇺 Warehouse-native Analytics compatible with Google Analytics and Matomo tracking protocols. Ingest into ClickHouse, BigQuery or CSV/Parquet on S3/GCS/local fs while maintaining complete control over your data.

• clickhouse
• ga4
• tracker
• bigquery

Code to download, process, and analyse the Online Encyclopedia of Integer Sequences

Enables search with Algolia in Hugo static sites

EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.

• dotnet
• dotnet-core
• etl
• etl-framework

Java importer skeleton for complicated, business-logic-heavy high-performance Neo4j imports directly from SQL databases, CSV files, etc.

• neo4j
• java
• neo4j-graphaware-framework

Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.

• postgres
• postgresql
• python
• infectious-disease

Internet Archive's Sparkling Data Processing Library

HBase adapters for Cascading