Repository Index

Parallel computing with task scheduling

• dask
• python
• pydata
• numpy

Blazing-fast, Neuro-symbolic Data-Wrangling toolkit

• csv
• data-wrangling
• opendata
• data-engineering

PySpark + Scikit-learn = Sparkit-learn

• scikit-learn
• apache-spark
• machine-learning
• distributed-computing

Data validation toolkit for assessing and monitoring data quality.

• data-validation
• tabular-data
• data-testing
• data-quality

A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.

📜 A CLI toolkit for extracting and working with your digital history

• chronicle
• chronicle-etl
• personal-data
• etl

A lightweight Reactive Streams Infrastructure Toolkit for Scala.

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

• spark
• hadoop
• webarchives
• analysis

EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.

• dotnet
• dotnet-core
• etl
• etl-framework

Develop streaming applications for IBM Streams in Python, Java & Scala.

• ibm-streams
• java
• scala
• python

Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.

• notebooks
• pyspark-notebook
• juypter-notebook
• spark

AsyncJob is an asynchronous queue job manager with light code, clear and speed. I hope so ! 😬

• async
• queue
• simple
• lightweight

An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.

• spark
• tweets
• twitter-data
• twitter-json

Quick integration toolkit for Ixian. Connects apps, IoT devices, AI Agents, and services to the Ixian Platform via REST APIs and message queues (MQTT/RabbitMQ). Provides developers with a simple bridge into secure Ixian communication and Client Discovery.

• api
• encryption
• integration
• iot

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.

• collaborative-editing
• conflict-resolution
• crdt
• data-engineering