Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Repository Index
47,133 repositories indexed
1–16 of 16 · Most stars
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
An orchestration platform for the development, production, and observation of data assets.
Incremental engine for long horizon agents 🌟 Star if you like it!
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
visual data prep powered by python
This repository is a getting started guide to Singer.
Reverse ETL for the code-first data stack
Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.
A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.
🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines
ETL scripts for EOS.
Data ingestion engine — Singer taps to Parquet in cloud buckets