Python Streaming DataFrames for Kafka
Repository Index
100,822 visible repositories in the index
49–72 of 108 · Most stars
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 427 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
- etl-pipeline
- etl-framework
- spark
- apache-spark
Hop Orchestration Platform
- java
- streaming
- hop
- apache
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.
- hadoop
- hive
- database
- clickhouse
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.
- blockchain
- blockchain-indexer
- ethereum
- etl
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
- elasticsearch
- go
- golang
- mongodb
Python Extract Transform and Load Tables of Data
Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
- apache-beam
- dataflow-templates
- google-cloud-dataflow
- google-cloud-storage
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
- integration
- reactive
- akka
- akka-stream
Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)
- search
- search-engine
- search-interface
- ocr
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
- logstash
- elk
- kafka
- elasticsearch
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
- transform
- etl
- xml
- json
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
Healthcare data infrastructure for next generation care delivery.
- api
- healthcare
- open-source
- soc2
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
The simple, stupid batch framework for Java
- java
- batch-processing
- batch
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
🌱 The ultimate solution for populating your MongoDB database.
Distributed Data Transfer Service for MySQL
- mysql
- data-transfer
- data-migration
- replication