Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–18 of 18 · Most stars
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
TranslatedPython scripts. Simulated login to Zhihu, web crawlers, working with Excel, WeChat official accounts, remote power-on
Python bindings for llama.cpp
A collection of important graph embedding, classification and representation learning papers with implementations.
- graph2vec
- classification-algorithm
- graph-kernels
- weisfeiler-lehman
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
Shell agnostic git based dotfiles package manager, written in Python.
Investor’s Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future?
Describe a thing, get a hologram, a 3D model and its parts: built by any AI agent, honestly.
- 3d-visualization
- agent-skills
- ai-agents
- claude-code
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.
- anomaly-detection
- bigquery
- cli
- data-engineering
🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines
- etl
- etl-framework
- etl-pipeline
- python
NetLytics is a Hadoop-powered framework for performing advanced analytics on various kinds of networks logs
ETL scripts for EOS.
- eos
- eosio
- etl
- apache-beam
Data ingestion engine — Singer taps to Parquet in cloud buckets