Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Repository Index
47,133 repositories indexed
1–24 of 36 · Most stars
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
Always know what to expect from your data.
- pipeline-tests
- dataquality
- datacleaning
- datacleaner
🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.
- xonsh
- devops
- iterm2
- data-engineering
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
- dataframe
- python
- bigdata
- tabular-data
The Open Source Feature Store for AI/ML
- machine-learning
- features
- ml
- big-data
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.
- python
- utilities
- cache
- data-structures
Python Stream Processing
- kafka-streams
- kafka
- python
- asyncio
🌊 Online machine learning in Python
- incremental-learning
- machine-learning
- python
- online-learning
Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Koalas: pandas API on Apache Spark
WebSocket and WAMP in Python for Twisted and asyncio
- python
- websocket
- real-time
- wamp
CLI task management & automation tool
- python
- build-tool
- build-automation
- task-runner
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Python Stream Processing. A Faust fork
A library for event sourcing in Python.
- domain-driven-design
- ddd
- eventsourcing
- cqrs
Python Streaming DataFrames for Kafka
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
MLBox is a powerful Automated Machine Learning python library.
- machine-learning
- auto-ml
- kaggle
- deep-learning
asyncio client for kafka
- asyncio
- kafka
- kafka-client
Python3 client for NATS
- asyncio
- nats
- python3
- cloud-native