47,133 repositories indexed
1–24 of 29 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
Migrate to PostgreSQL in a single command!
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
🌱 The ultimate solution for populating your MongoDB database.
Distributed Data Transfer Service for MySQL
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰
📜 A CLI toolkit for extracting and working with your digital history
go-etl is a toolset for data extraction, transformation and loading.
Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.
Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.
A high performance data transfer tool that can be used to transfer data from Bitcoin Core blockchain files to a SQL Server database
This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.
Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.
Code to download, process, and analyse the Online Encyclopedia of Integer Sequences