Repository Index

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

• bloom
• deep-learning
• distributed-systems
• language-models

Build, Manage and Deploy AI/ML Systems

• machine-learning
• model-management
• ai
• ml

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

• mlops
• machine-learning
• data-science
• production-ready

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

• dataops
• dataops-platform
• elt
• open-source

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

• hpc
• workflow
• workflow-automation
• workflow-management

The data transformation tool the open-source community deserves.

• data-engineering
• duckdb

Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.

• apache-iceberg
• apache-spark
• data-engineering
• delta-lake

High performance Kafka consumer for InfluxDB. Supports collectd message formats.

• kafka-influxdb
• kafka-consumer
• logstash
• influxdb

BeanQueue, a lightweight Python task queue framework based on SQLAlchemy, PostgreSQL SKIP LOCKED queries and NOTIFY / LISTEN statements.

• postgresql
• python
• queue
• sqlalchemy

Blazingly fast data comparison tool for Python, powered by Rust. Compare massive CSV/Parquet datasets instantly.

• csv
• data-engineering
• data-quality
• diff

ETL scripts for EOS.

• eos
• eosio
• etl
• apache-beam