data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
- data
- python
- data-engineering
- data-lake
Repository Index
67,262 visible repositories in the index
25–48 of 136 · Most stars
TranslatedDataX integrated with a visual page: select a data source and generate a data synchronization task with one click. Supports data sources such as RDBMS, Hive, HBase, ClickHouse, and MongoDB, and batch creation of RDBMS data synchronization tasks. Integrates an open-source scheduling system, and supports distributed and incremental data synchronization, real-time viewing of running logs, monitoring of executor resources, KILL of running processes, encryption of data source information, and more.
Neo4j graph construction from unstructured data using LLMs
- data-import
- genai
- graph
- graph-rag
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
- data-integration
- clickhouse
- golang
- bigquery
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
RedisShake is a Valkey/Redis data processing and migration tool.
- redis
- redis-cluster
- redis-proxy
A system for agentic LLM-powered data processing and ETL
Cartography is a Python tool that pulls infrastructure assets and their relationships into a Neo4j graph database.
Sync MySQL data into elasticsearch
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
A data integration framework
The open source repository for Electricity Maps data parsers that powers the world's most comprehensive electricity data platform
- sustainability
- data-visualization
- climate-change
- hacktoberfest
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
The Data Transfer Project makes it easy for platforms to build interoperable user data portability features. We are establishing a common framework, including data models and protocols, to enable direct transfer of data both into and out of participating online service providers.
- data
- transfer
- portability
- data-portability
A Dockerized python Script to fetch Garmin health data and populate that in a InfluxDB Database, for visualization long term health trends with Grafana
- garmin
- garmin-api
- garmin-connect
- garmin-data
Contentlayer turns your content into data - making it super easy to import MD(X) and CMS content in your app
Sync data from one Postgres database to another
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
- machine-learning
- convolutional-networks
- embedding-vectors
- embeddings
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
LFX Community Data Platform (CDP)
- community
- cdp
- open-source
Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
- garmin
- python
- health
- database
Scalable and efficient data transformation framework - backwards compatible with dbt.
Django application and library for importing and exporting data with admin integration.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native