Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
All-in-one IoT Platform - Device management, data collection, processing and visualization.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
High-Performance server for NATS.io, the cloud and edge native messaging system.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
GoDS (Go Data Structures) - Sets, Lists, Stacks, Maps, Trees, Queues, and much more
- go
- golang
- data-structure
- map
The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
- metadata
- datadiscovery
- dataquality
- data-profiling
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
Graph-Native Infrastructure for Context and Accountable AI Systems
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
- workflow-schedule
- azkaban
- airflow
- task-scheduler
Node-based Visual Programming Toolbox
OpenRefine is a free, open source power tool for working with messy data and improving it
CMAK is a tool for managing Apache Kafka clusters
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
Evolution API is an open-source WhatsApp integration API
Building event-driven applications the easy way in Go.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.
Fancy stream processing made operationally mundane
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
Apache Beam is a unified programming model for Batch and Streaming data processing.
Kafka library in Go