Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
109,091 visible repositories in the index
1–24 of 29 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Graph-Native Semantic Layer for Trustworthy Enterprise AI
🧙 Build, run, and manage data pipelines for integrating and transforming data.
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
A system for agentic LLM-powered data processing and ETL
Cartography is a Python tool that pulls infrastructure assets and their relationships into a Neo4j graph database.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
The open source repository for Electricity Maps data parsers that powers the world's most comprehensive electricity data platform
A Dockerized python Script to fetch Garmin health data and populate that in a InfluxDB Database, for visualization long term health trends with Grafana
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Scalable and efficient data transformation framework - backwards compatible with dbt.
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Extract Transform Load for Python 3.5+
Python Streaming DataFrames for Kafka
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Python Extract Transform and Load Tables of Data