SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Repository Index
47,133 repositories indexed
1–24 of 25 · Most stars
TranslatedDataX is the open-source version of Alibaba Cloud DataWorks Data Integration.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
- data
- python
- data-engineering
- data-lake
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
- data
- ingestion
- replication
- management
Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.
- dataprep
- data-science
- datapreparation
- dataconnector
Data processing & ETL framework for Ruby
- etl
- etl-ruby
- ruby
- data
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
- transform
- etl
- xml
- json
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!
- code-generator
- etl
- dotnet-core
- dotnet
Distributed Data Transfer Service for MySQL
- mysql
- data-transfer
- data-migration
- replication
The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
📜 A CLI toolkit for extracting and working with your digital history
- chronicle
- chronicle-etl
- personal-data
- etl
This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.
- magento
- import
- products
- categories
Doc manager for Neo4j
🇪🇺 Warehouse-native Analytics compatible with Google Analytics and Matomo tracking protocols. Ingest into ClickHouse, BigQuery or CSV/Parquet on S3/GCS/local fs while maintaining complete control over your data.
- clickhouse
- ga4
- tracker
- bigquery
Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.
EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.
- dotnet
- dotnet-core
- etl
- etl-framework
Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.
- postgres
- postgresql
- python
- infectious-disease
Internet Archive's Sparkling Data Processing Library
Real-time GTFS database loader and ORM library.
Jmeter JTL parsing with Logstash for Elasticseacrh and Influxdb ...
- jmeter
- logstash
- filebeat
- elasticsearch