Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
47,133 repositories indexed
1–24 of 103 · Most stars
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
Apache Hadoop
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
- metadata
- datadiscovery
- dataquality
- data-profiling
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
The leader in Customer Data Infrastructure
- snowplow
- analytics
- data
- data-pipeline
🍴 Web applications made easy. Since 2011.
Migrate to PostgreSQL in a single command!
- postgresql
- migration
- mysql
- mssql
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
- data
- python
- data-engineering
- data-lake
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
- mlops
- machine-learning
- data-science
- production-ready
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
🪽 An open, easy, fast, reliable and battery-efficient solution for real-time communications
- api
- web-api
- server-sent-events
- subscriptions
The AsyncAPI specification allows you to create machine-readable definitions of your asynchronous APIs.
- asyncapi
- reactive
- specification
- websockets
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
Blazing-fast, Neuro-symbolic Data-Wrangling toolkit
- csv
- data-wrangling
- opendata
- data-engineering
The open source and enterprise-ready webhooks service 🦀
- webhooks
- rust
- webhook-server
- webhook-service
World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
- datalake
- lakehouse
- metadata
- federated-query