Repository Index

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python

Logstash - transport and process your logs, events, or other data

• etl-framework
• streaming
• logging
• java

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time

Fancy stream processing made operationally mundane

• message-queue
• stream-processing
• streaming-data
• message-bus

Privacy and Security focused Segment-alternative, in Golang and React

• privacy
• warehouse-management
• data-warehouse
• customer-data

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

• etl
• sql
• bigquery
• cloud-native

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

• data
• ingestion
• replication
• management

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering

Apache InLong - a one-stop, full-scenario integration framework for massive data

• inlong
• one-stop-service
• data-streaming
• event-streaming

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

• transform
• etl
• xml
• json

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

• go
• data-integration
• etl
• data-pipeline

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

• java
• smooks
• enterprise-integration
• etl