Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–21 of 21 · Most stars
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
- oneliner-commands
- bash
- data-processing
- linux-administration
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.
- json
- yaml
- configuration
- selector
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
- cloud-native
- streaming
- rust
- real-time
A light-weight, flexible, and expressive statistical data testing library
- pandas
- validation
- schema
- dataframes
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
Concurrent and multi-stage data ingestion and data processing with Elixir
- elixir
- genstage
- data-ingestion
- data-processing
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Stream Processing and Complex Event Processing Engine
- streaming-data-processing
- cep
- stream-processing
- complex-event-processing
Python Streaming DataFrames for Kafka
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
👾~ music, eternal ~ 👾
- glsl
- node-based
- music
- 3d-graphics
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
- data-science
- data-processing
- python
- apache-spark
Public tracker for Scramjet Cloud Platform, a platform that bring data from many environments together.
- data-stream
- event-stream
- transformations
- reactive-programming
Read and write Excel (.xlsx, .xls), CSV, and ODS files in Java with one line of code. Annotation-based mapping, SAX streaming, built-in validation, 19 type converters, cell styling. A high-performance alternative to Apache POI.
- annotation
- apache-poi
- csv
- data-processing
Data policy IN, dynamic view OUT: PACE is the Policy As Code Engine. It helps you to programatically create and apply a data policy to a processing platform like Databricks, Snowflake or BigQuery (or plain 'ol Postgres, even!) with definitions imported from Collibra, Datahub, ODD and the like.
- data-catalog
- data-processing
- policy-enforcement
- bigquery
High performant ETL engine written in TypeScript
- data-processing
- etl
- etl-framework
- streaming-data