47,133 repositories indexed
1–19 of 19 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
visual data prep powered by python
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
The data transformation tool the open-source community deserves.
go-etl is a toolset for data extraction, transformation and loading.
:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
Scramjet Platform is a distributed runtime supervisor that can run data processing programs called Sequences and manage local resources on any Linux server, Docker on small edge servers, and even large-scale Kubernetes clusters in the cloud or datacenters without exposing network ports to the public internet.
EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.