Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 41 · Most stars
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Cross-platform, customizable ML solutions for live and streaming media.
- mediapipe
- c-plus-plus
- computer-vision
- deep-learning
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
- pandas-profiling
- pandas-dataframe
- statistics
- jupyter-notebook
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
- weak-supervision
- data-cleaning
- data-quality
- noisy-labels
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
The Open Source Feature Store for AI/ML
- machine-learning
- features
- ml
- big-data
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
- mlops
- machine-learning
- data-science
- production-ready
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
Machine learning platform for Web developers
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Elyra extends JupyterLab with an AI centric approach.
- notebooks
- notebook-jupyter
- jupyterlab
- jupyterlab-extensions
A minimal benchmark for scalability, speed and accuracy of commonly used open source implementations (R packages, Python scikit-learn, H2O, xgboost, Spark MLlib etc.) of the top machine learning algorithms for binary classification (random forests, gradient boosted trees, deep neural networks etc.).
Python Streaming DataFrames for Kafka
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
MLBox is a powerful Automated Machine Learning python library.
- machine-learning
- auto-ml
- kaggle
- deep-learning
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream