Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Repository Index
47,133 repositories indexed
1–7 of 7 · Most stars
Python best practices guidebook, written for humans.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
BullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL
Apache Beam is a unified programming model for Batch and Streaming data processing.
Cloud-native genomic dataframes and batch computing
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.