Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
- go
- data-integration
- etl
- data-pipeline
Repository Index
47,133 repositories indexed
25–44 of 44 · Most stars
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
Analyse GTFS Schedule data by importing it into a PostgreSQL database.
Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.
🇪🇺 Warehouse-native Analytics compatible with Google Analytics and Matomo tracking protocols. Ingest into ClickHouse, BigQuery or CSV/Parquet on S3/GCS/local fs while maintaining complete control over your data.
- clickhouse
- ga4
- tracker
- bigquery
Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.
Code to download, process, and analyse the Online Encyclopedia of Integer Sequences
Insert incoming MQTT values into InfluxDB. Follows mqtt-smarthome architecture.
EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.
- dotnet
- dotnet-core
- etl
- etl-framework
Interfaces ham radio APRS-IS servers and saves packet data into an influxdb database
Internet Archive's Sparkling Data Processing Library
R COBOL DI (Data Integration) Package : Import COBOL CopyBook data files directly into R as properly structured data frames.
- r
- data-science
- dataframe
- data-frames
Analyse GTFS Schedule data by importing it into a DuckDB database.
Parse cowrie honeypot logs into a neo4j database
A library and supporting script for pulling data from vSphere and inserting it into InfluxDB
Reads EVE files into SQL as well as search stored data.
Sitecore Content Hub Asset Importer. Allows importing image assets from various data sources into Sitecore Content Hub.