Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
105,102 visible repositories in the index
1–24 of 27 · Most stars
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Neo4j graph construction from unstructured data using LLMs
- data-import
- genai
- graph
- graph-rag
Sync data from one Postgres database to another
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
- garmin
- python
- health
- database
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
- data
- data-analysis
- data-engineering
- data-integration
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- sql
- redpanda
- single-binary
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
The ToroDB solution to provide better analytics on top of MongoDB and make it easier to migrate from MongoDB to SQL
Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.
- apis
- data
- rust
- sql
TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.
- blockchain
- blockchain-indexer
- ethereum
- etl
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
- logstash
- elk
- kafka
- elasticsearch
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
- spark
- neo4j-connector
- cypher
- neo4j-driver
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰
- csv-import
- data-import
- data-mapping
- data-review
A decision-model-based ontology & knowledge-graph builder — builds governed knowledge graphs from your documents, every fact gated and traceable to its evidence.
- graphrag
- information-extraction
- knowledge-graph
- llm
A high performance data transfer tool that can be used to transfer data from Bitcoin Core blockchain files to a SQL Server database
Export your database from Lovable Cloud to Supabase
Java importer skeleton for complicated, business-logic-heavy high-performance Neo4j imports directly from SQL databases, CSV files, etc.
- neo4j
- java
- neo4j-graphaware-framework