Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 49 · Most stars
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.
- chatbot
- chatgpt-api
- chatglm
- claude
Always know what to expect from your data.
- pipeline-tests
- dataquality
- datacleaning
- datacleaner
Python SQL Parser and Transpiler
- transpiler
- sql
- python
- parser
🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.
- xonsh
- devops
- iterm2
- data-engineering
The Open Source Feature Store for AI/ML
- machine-learning
- features
- ml
- big-data
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.
- python
- utilities
- cache
- data-structures
the portable Python dataframe library
a little task queue for python
- redis
- python
- task-queue
- queue
Event-driven networking engine written in Python.
- python
- twisted
- async
- async-python
Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Messaging library for Python.
- celery
- messaging
- python
- message-queue
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
CLI task management & automation tool
- python
- build-tool
- build-automation
- task-runner
Elyra extends JupyterLab with an AI centric approach.
- notebooks
- notebook-jupyter
- jupyterlab
- jupyterlab-extensions
Lightweight and extensible compatibility layer between dataframe libraries!
Python Streaming DataFrames for Kafka
Distributed Deep learning with Keras & Spark
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark