47,133 repositories indexed
1–24 of 38 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
Fancy stream processing made operationally mundane
The leader in Customer Data Infrastructure
Migrate to PostgreSQL in a single command!
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
Blazing-fast, Neuro-symbolic Data-Wrangling toolkit
Self-hosted platform to unify wearable health data through one AI-ready API.
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Embulk: Pluggable Bulk Data Loader.
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
Amazon Redshift Database Loader implemented in AWS Lambda
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
The data transformation tool the open-source community deserves.
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
Blockchain Indexing Engine
go-etl is a toolset for data extraction, transformation and loading.
Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.
A high performance data transfer tool that can be used to transfer data from Bitcoin Core blockchain files to a SQL Server database