Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
96,838 visible repositories in the index
1–24 of 101 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
All-in-one IoT Platform - Device management, data collection, processing and visualization.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Logstash - transport and process your logs, events, or other data
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Node-based Visual Programming Toolbox
Incremental engine for long horizon agents 🌟 Star if you like it!
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
Distributed Application Framework for .NET
Broadcast, Presence, and Postgres Changes via WebSockets
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
A lightweight data processing framework built on DuckDB and 3FS.
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
A light-weight, flexible, and expressive statistical data testing library
A data integration framework
Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.
AI Infra / AI Orchestration / AI Control Plane
The Data Transfer Project makes it easy for platforms to build interoperable user data portability features. We are establishing a common framework, including data models and protocols, to enable direct transfer of data both into and out of participating online service providers.