Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 35 · Most stars
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
The leader in Customer Data Infrastructure
- snowplow
- analytics
- data
- data-pipeline
Migrate to PostgreSQL in a single command!
- postgresql
- migration
- mysql
- mssql
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Blazing-fast, Neuro-symbolic Data-Wrangling toolkit
- csv
- data-wrangling
- opendata
- data-engineering
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
Import CSV and JSON into PostgreSQL the easy way
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
- transform
- etl
- xml
- json
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰
- csv-import
- data-import
- data-mapping
- data-review
Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.
📜 A CLI toolkit for extracting and working with your digital history
- chronicle
- chronicle-etl
- personal-data
- etl
go-etl is a toolset for data extraction, transformation and loading.
Scalable Full History API Solution for Antelope (former EOSIO) based blockchains
- antelope
- api
- big-data
- blockchain
Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.
This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.
- magento
- import
- products
- categories