Repository Index

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

• aws
• gcp
• azure
• sql

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

• dataops
• dataops-platform
• elt
• open-source

Clean APIs for data cleaning. Python implementation of R package Janitor

• pandas
• dataframe
• data
• cleaning-data

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

• change-data-capture
• data
• data-collection
• data-engineering

Data ingestion engine — Singer taps to Parquet in cloud buckets

• data
• data-engineering
• data-lake
• duckdb