Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
47,133 repositories indexed
1–24 of 49 · Most stars
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Apache Pulsar - distributed pub-sub messaging system
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
- weak-supervision
- data-cleaning
- data-quality
- noisy-labels
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern
Event-driven networking engine written in Python.
- python
- twisted
- async
- async-python
Distributed stream processing engine in Rust
Where data access meets operational intelligence
- mysql
- postgresql
- ai-agent
- data-catalog
Orchestrate end-to-end encryption, cryptographic identities, mutual authentication, and authorization policies between distributed applications – at massive scale.
- messaging
- authentication
- key-management
- identity
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Examples and Tutorials of Event Sourcing in .NET
- event-sourcing
- cqrs
- dotnet-core
- example
Utils for streaming large files (S3, HDFS, gzip, bz2...)
- python
- s3
- hdfs
- webhdfs
High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
- postgresql
- node
- worker
- job-queue
Ruby and Rails efficient Kafka processing framework
- karafka-framework
- kafka
- kafka-client
- kafka-clients
A lightweight stream processing library for Go
Composable concurrency toolkit for Go
- concurrency
- channels
- go
- golang
🚀Gev is a lightweight, fast non-blocking TCP network library / websocket server based on Reactor mode. Support custom protocols to quickly and easily build high-performance servers.
- epoll-tcp-server
- kqueue
- golang
- reactor
Lightweight and extensible compatibility layer between dataframe libraries!
Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
🦎 A lightweight, streaming-native, multi-protocol gateway for AI, APIs, and real-time data. Route LLM requests across providers, govern MCP tools, and connect Kafka, MQTT, and HTTP through one high-performance engine with shared security and observability.
A Ruby implementation of an Event Store based on Active Record
- ruby
- event-sourcing
- rails
- ddd
Jupyter magics and kernels for working with remote Spark clusters