Repository Index

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

• java
• smooks
• enterprise-integration
• etl

Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.

go-etl is a toolset for data extraction, transformation and loading.

• etl
• mysql
• go
• postgresql

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

Event-driven OneLake pipeline with automated Upsert merge and Python reporting

• activator
• data-engineering
• data-factory
• data-pipeline

RDF-Centric Map/Reduce Framework and Freebase data conversion tool

Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.

• gis
• postgis
• geoserver
• golang

A Substrate indexing framework

• substrate

Export your database from Lovable Cloud to Supabase

Algorand's data pipeline framework.

Reverse ETL for the code-first data stack

• bigquery
• cli
• data-engineering
• dbt

EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.

• dotnet
• dotnet-core
• etl
• etl-framework

A simple data project for Steam data

• steam
• steam-data
• steam-api
• python

Interfaces ham radio APRS-IS servers and saves packet data into an influxdb database

Java importer skeleton for complicated, business-logic-heavy high-performance Neo4j imports directly from SQL databases, CSV files, etc.

• neo4j
• java
• neo4j-graphaware-framework

High performant ETL engine written in TypeScript

• data-processing
• etl
• etl-framework
• streaming-data

R COBOL DI (Data Integration) Package : Import COBOL CopyBook data files directly into R as properly structured data frames.

• r
• data-science
• dataframe
• data-frames

ETL scripts for EOS.

• eos
• eosio
• etl
• apache-beam