Repository Index

Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

Apache InLong - a one-stop, full-scenario integration framework for massive data

• inlong
• one-stop-service
• data-streaming
• event-streaming

visual data prep powered by python

• data
• data-pipelines
• etl
• structured-data

Import CSV and JSON into PostgreSQL the easy way

Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.

• data-science
• data-visualization
• visualization
• data-analysis

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream

The swiss army knife of healthcare integration.

• mirth
• mirth-connect
• hl7
• healthcare

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

• transform
• etl
• xml
• json

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

• change-data-capture
• data
• data-collection
• data-engineering

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

• csv
• parser
• writer
• reader

Healthcare data infrastructure for next generation care delivery.

• api
• healthcare
• open-source
• soc2

The MongoDB Spark Connector

• spark
• spark-packages
• mongo-spark
• mongodb

:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.

• porter
• data-import
• framework
• data-transformation

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!

• code-generator
• etl
• dotnet-core
• dotnet

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

Graph-based security analysis for everyone

• gcp
• graph
• aws
• security

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs

• spark
• neo4j-connector
• cypher
• neo4j-driver

Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.

Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.

• garmin
• typescript
• mqtt
• ble

📜 A CLI toolkit for extracting and working with your digital history

• chronicle
• chronicle-etl
• personal-data
• etl

Blockchain Indexing Engine

• polkadot
• substrate
• postgres
• indexing

A plugin to push to Algolia based on graphQl queries

• gatsby
• algolia
• search
• static-site-generator

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving