Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
103,113 visible repositories in the index
1–16 of 16 · Most stars
Graph-Native Semantic Layer for Trustworthy Enterprise AI
- ai
- ai-governance
- artificial-intelligence
- context-engineering
🧙 Build, run, and manage data pipelines for integrating and transforming data.
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
- ai-agents
- data-engineering
- document-ai
- generative-ai
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
- data
- python
- data-engineering
- data-lake
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Scalable and efficient data transformation framework - backwards compatible with dbt.
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
- claude-code
- codex
- data-processing
- harness-engineering
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
Python Streaming DataFrames for Kafka
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
- etl-pipeline
- etl-framework
- spark
- apache-spark
Concurrent Python made simple
- asyncio
- concurrency
- python
- threading
The data transformation tool the open-source community deserves.
- data-engineering
- duckdb
- data-lineage
- data-pipeline
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
ETL scripts for EOS.
- eos
- eosio
- etl
- apache-beam
Data ingestion engine — Singer taps to Parquet in cloud buckets