Repository Index

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

• kedro
• hacktoberfest
• mlops
• python

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering

MLeap: Deploy ML Pipelines to Production

• scikit-learn
• spark
• data-pipelines
• transformers