Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
132,241 visible repositories in the index
1–24 of 32 · Most stars
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
The open source repository for Electricity Maps data parsers that powers the world's most comprehensive electricity data platform
- sustainability
- data-visualization
- climate-change
- hacktoberfest
Contentlayer turns your content into data - making it super easy to import MD(X) and CMS content in your app
Django application and library for importing and exporting data with admin integration.
Spring Batch is a framework for writing batch applications using Java and Spring
- batch
- batch-processing
- java
- spring
Self-hosted platform to unify wearable health data through one AI-ready API.
- wearables
- apple-health
- apple-watch
- fitness
A high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary.
- logical-replication
- postgres
- replication
- rust
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
- unstructured-data
- low-code
- developer-tools
- etl
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- sql
- redpanda
- single-binary
Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.
- dataprep
- data-science
- datapreparation
- dataconnector
Scalable data pre processing and curation toolkit for LLMs
- data-curation
- llm
- data
- data-prep
Python Streaming DataFrames for Kafka
Support agile Ontology DataOps Based on Flink, DataX and Flink-CDC with Web-UI
Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)
- search
- search-engine
- search-interface
- ocr
simple database import/export app
MetaForge is integrated software for data transformation, integration, and visualization. Support a variety of data sources, file types, big data related database, relational database, NoSQL database, etc. Through the software can realize the management of multiple data sources, the data under the source of various operations conversion, making dat
- database
- clickhouse
- mysql
- kyuubi
Graph+Semantics: Import/Export RDF from Neo4j. SHACL Validation, Model mapping and more.... If you like it, please ★ ⇧
- rdf
- neo4j
- semantics
- ontologies
Open source project for data preparation for GenAI applications
- data-preparation
- finetuning
- llm
- llmapps
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
🌱 The ultimate solution for populating your MongoDB database.
The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl