Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–18 of 18 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Database Subsetting and Relational Data Browsing Tool.
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
- go
- data-integration
- etl
- data-pipeline
Amazon Redshift Database Loader implemented in AWS Lambda
RDF-Centric Map/Reduce Framework and Freebase data conversion tool
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.
- postgres
- postgresql
- python
- infectious-disease
R COBOL DI (Data Integration) Package : Import COBOL CopyBook data files directly into R as properly structured data frames.
- r
- data-science
- dataframe
- data-frames
Reads EVE files into SQL as well as search stored data.