Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
108,094 visible repositories in the index
1–24 of 29 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Graph-Native Semantic Layer for Trustworthy Enterprise AI
🧙 Build, run, and manage data pipelines for integrating and transforming data.
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
A system for agentic LLM-powered data processing and ETL
Cartography is a Python tool that pulls infrastructure assets and their relationships into a Neo4j graph database.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
The open source repository for Electricity Maps data parsers that powers the world's most comprehensive electricity data platform
A Dockerized python Script to fetch Garmin health data and populate that in a InfluxDB Database, for visualization long term health trends with Grafana
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Self-hosted platform to unify wearable health data through one AI-ready API.
Python CLI utility and library for manipulating SQLite databases
Scalable data pre processing and curation toolkit for LLMs
Python Streaming DataFrames for Kafka
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync
The data transformation tool the open-source community deserves.
A decision-model-based ontology & knowledge-graph builder — builds governed knowledge graphs from your documents, every fact gated and traceable to its evidence.
Code to download, process, and analyse the Online Encyclopedia of Integer Sequences