Repository Index

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl

Upserts, Deletes And Incremental Processing on Big Data.

• hudi
• apachehudi
• datalake
• bigdata

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

• workflow
• governance
• azkaban
• davinci

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.

• scala
• sqs-interface
• elasticmq
• amazon-sqs

Highly optimized, extremely modular and very flexible XMPP/Jabber server

• tigase
• xmpp
• server
• java

Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.

Data Quality and Observability platform for the whole data lifecycle, from profiling new data sources to full automation with Data Observability. Configure data quality checks from the UI or in YAML files, let DQOps run the data quality checks daily to detect data quality issues.

• data-ops
• data-quality
• data-quality-checks
• data-quality-measurement

Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.

Emissary is simple, yet 🗲FAST🗲 messaging library for decoupling messages (requests and events) and message handlers 🚀

• java
• jvm
• messaging
• requests

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.

Multi-core Window-Based Stream Processing Engine

• llvm
• incremental-computation
• aggregation
• stream-processing

Scramjet Platform is a distributed runtime supervisor that can run data processing programs called Sequences and manage local resources on any Linux server, Docker on small edge servers, and even large-scale Kubernetes clusters in the cloud or datacenters without exposing network ports to the public internet.

• nodejs
• stream-processing
• transform-as-a-service
• real-time