Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–14 of 14 · Most stars
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
- apache-spark
- big-data
- connected-components
- dataframe
Data structure and algorithm library for go, designed to provide functions similar to C++ STL
.NET Stream Processing Library for Apache Kafka 🚀
- kafka-streams-dotnet
- kafka-streams
- kafka-streams-net
- kafka
Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.
- data-analytics
- data-cleaning
- data-cleansing
- data-engineering
RabbitMQ wire tap and swiss army knife
- rabbitmq
- tap
- wiretap
- golang
A plugin to push to Algolia based on graphQl queries
- gatsby
- algolia
- search
- static-site-generator
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.
Distributed batch data processing framework
Enables search with Algolia in Hugo static sites
Algorand's data pipeline framework.
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Partition (W)ARC Files by MIME Type and Year
- web-archiving
- webarchive
- hadoop
- warc