Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
107,098 visible repositories in the index
1–24 of 30 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
🧙 Build, run, and manage data pipelines for integrating and transforming data.
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
A system for agentic LLM-powered data processing and ETL
Cartography is a Python tool that pulls infrastructure assets and their relationships into a Neo4j graph database.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
A Dockerized python Script to fetch Garmin health data and populate that in a InfluxDB Database, for visualization long term health trends with Grafana
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
Scalable and efficient data transformation framework - backwards compatible with dbt.
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.
Python CLI utility and library for manipulating SQLite databases
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync
Python Extract Transform and Load Tables of Data