Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
103,113 visible repositories in the index
1–24 of 27 · Most stars
Graph-Native Semantic Layer for Trustworthy Enterprise AI
- ai
- ai-governance
- artificial-intelligence
- context-engineering
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
- ai-agents
- data-engineering
- document-ai
- generative-ai
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
- data
- python
- data-engineering
- data-lake
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Scalable and efficient data transformation framework - backwards compatible with dbt.
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
- data
- data-analysis
- data-engineering
- data-integration
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
- claude-code
- codex
- data-processing
- harness-engineering
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Python Streaming DataFrames for Kafka
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 427 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
- etl-pipeline
- etl-framework
- spark
- apache-spark
Concurrent Python made simple
- asyncio
- concurrency
- python
- threading
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
- change-data-capture
- data
- data-collection
- data-engineering
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
- go
- data-integration
- etl
- data-pipeline
The data transformation tool the open-source community deserves.
- data-engineering
- duckdb
- data-lineage
- data-pipeline
Event-driven OneLake pipeline with automated Upsert merge and Python reporting
- activator
- data-engineering
- data-factory
- data-pipeline