Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
86,076 visible repositories in the index
1–12 of 12 · Most stars
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
- machine-learning
- convolutional-networks
- embedding-vectors
- embeddings
Scalable and efficient data transformation framework - backwards compatible with dbt.
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
- unstructured-data
- low-code
- developer-tools
- etl
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Python Streaming DataFrames for Kafka
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
- etl-pipeline
- etl-framework
- spark
- apache-spark
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
Interfaces ham radio APRS-IS servers and saves packet data into an influxdb database
ETL scripts for EOS.
- eos
- eosio
- etl
- apache-beam