Repository Index

TranslatedAlibaba's MySQL binlog incremental subscription & consumption component

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

• metadata
• datadiscovery
• dataquality
• data-profiling

Logstash - transport and process your logs, events, or other data

• etl-framework
• streaming
• logging
• java

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

• change-data-capture
• kafka-connect
• apache-kafka
• debezium

Fancy stream processing made operationally mundane

• message-queue
• stream-processing
• streaming-data
• message-bus

A network event stream processing system, in Clojure.

• riemann
• events
• clojure
• monitoring

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query

Keep passwords and other sensitive information out of your chat logs and inboxes.

• privacy
• onetime
• secrets
• chat

StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.

• security
• kinesis
• serverless
• terraform

Project Flogo is an open source ecosystem of opinionated event-driven capabilities to simplify building efficient & modern serverless functions, microservices & edge apps.

• flogo
• iot
• integration
• golang

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

• data
• spark
• java
• iceberg

Ultra quick message queue and streaming server

• message-queue
• server
• amqp
• background-jobs

A distributed MySQL binlog storage system built on Raft

• mysql
• raft
• golang
• mysql-binlog

Automatic data change tracking for PostgreSQL

• audit-log
• audit-trail
• change-data-capture
• change-tracking

Enhanced PostgreSQL logical replication

• postgresql
• logical-replication
• golang

Phoebus is a distributed framework for large scale graph processing written in Erlang.

Streaming Postgres logical replication changes atleast-once over websockets

• postgresql
• logical-replication
• write-ahead-log
• websockets

High performance Kafka consumer for InfluxDB. Supports collectd message formats.

• kafka-influxdb
• kafka-consumer
• logstash
• influxdb

Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.

A Kafka log inspired in-memory and append-only data structure

A refined core library for D programming language. The module has concurrency / collections / event / io / logging / text / serialization and more.

• event-driven
• asynchronous
• io
• concurrency

Realistic synthetic events for testing, demos, and pipelines — streamed live or generated in bulk

• clickhouse
• data-engineering
• data-generation
• devops