Repository Index

Logstash - transport and process your logs, events, or other data

• etl-framework
• streaming
• logging
• java

Migrate to PostgreSQL in a single command!

• postgresql
• migration
• mysql
• mssql

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

• aws
• gcp
• azure
• sql

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

• data
• ingestion
• replication
• management

Apache Spark to Apache Cassandra connector

• cassandra
• scala
• spark

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

The MongoDB Spark Connector

• spark
• spark-packages
• mongo-spark
• mongodb

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs

• spark
• neo4j-connector
• cypher
• neo4j-driver

Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.

Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.

• garmin
• typescript
• mqtt
• ble

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

Analyse GTFS Schedule data by importing it into a PostgreSQL database.

• gtfs
• postgresql
• postgres
• sql

Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.

This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.

• magento
• import
• products
• categories

Internet Archive's Sparkling Data Processing Library

R COBOL DI (Data Integration) Package : Import COBOL CopyBook data files directly into R as properly structured data frames.

• r
• data-science
• dataframe
• data-frames

Analyse GTFS Schedule data by importing it into a DuckDB database.

• duckdb
• gtfs
• gtfs-schedule
• gtfs-static

Peercoin blockchain parser

Data ingestion engine — Singer taps to Parquet in cloud buckets

• data
• data-engineering
• data-lake
• duckdb

A library and supporting script for pulling data from vSphere and inserting it into InfluxDB