Repository Index

Polo travels through your database and creates sample snapshots so you can work with real world data in development.

• database
• snapshot
• productivity
• activerecord

The MongoDB Spark Connector

• spark
• spark-packages
• mongo-spark
• mongodb

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

• go
• data-integration
• etl
• data-pipeline

🌱 The ultimate solution for populating your MongoDB database.

• mongodb
• mongo
• db
• database

Distributed Data Transfer Service for MySQL

• mysql
• data-transfer
• data-migration
• replication

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

Graph-based security analysis for everyone

• gcp
• graph
• aws
• security

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs

• spark
• neo4j-connector
• cypher
• neo4j-driver

Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.

Apache HBase Connectors

• database
• java
• hbase

A data migration and visualization command line gem in Ruby

High performance Kafka consumer for InfluxDB. Supports collectd message formats.

• kafka-influxdb
• kafka-consumer
• logstash
• influxdb

Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.

• garmin
• typescript
• mqtt
• ble

AtlasMap project repository

• atlasmap

Blockchain Indexing Engine

• polkadot
• substrate
• postgres
• indexing

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.

This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.

• magento
• import
• products
• categories

🇪🇺 Warehouse-native Analytics compatible with Google Analytics and Matomo tracking protocols. Ingest into ClickHouse, BigQuery or CSV/Parquet on S3/GCS/local fs while maintaining complete control over your data.

• clickhouse
• ga4
• tracker
• bigquery

Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.

• gis
• postgis
• geoserver
• golang

Insert incoming MQTT values into InfluxDB. Follows mqtt-smarthome architecture.

• mqtt
• influxdb
• smarthome
• grafana

Open source industrial IoT connectivity gateway.

• iotsharp
• iot
• modbus
• iotgateway

Collecting, shape-enhancing, validating, fixing and (partially) merging GTFS feeds