Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–23 of 23 · Most stars
Logstash - transport and process your logs, events, or other data
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Database Subsetting and Relational Data Browsing Tool.
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
Polo travels through your database and creates sample snapshots so you can work with real world data in development.
- database
- snapshot
- productivity
- activerecord
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
Amazon Redshift Database Loader implemented in AWS Lambda
The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
High performance Kafka consumer for InfluxDB. Supports collectd message formats.
- kafka-influxdb
- kafka-consumer
- logstash
- influxdb
A plugin to push to Algolia based on graphQl queries
- gatsby
- algolia
- search
- static-site-generator
Bash script to export android content providers data to csv
Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
Open source industrial IoT connectivity gateway.
Collecting, shape-enhancing, validating, fixing and (partially) merging GTFS feeds
Live import all your Google Sheets to your data warehouse
- data-warehouse
- data-vault
- etl
- google-sheets
Jmeter JTL parsing with Logstash for Elasticseacrh and Influxdb ...
- jmeter
- logstash
- filebeat
- elasticsearch
A tool to grab AVL data from the database and push it to the ddot-updates service