Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Repository Index
47,133 repositories indexed
25–48 of 225 · Most stars
Turns Data and AI algorithms into production-ready web applications in no time.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Memory layer for AI Agents. Replace complex RAG pipelines with a serverless, single-file memory layer. Give your agents instant retrieval and long-term memory.
Workflow Engine for Kubernetes
- workflow
- kubernetes
- argo
- dag
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Unified framework for building enterprise RAG pipelines with small, specialized models
- parsing
- retrieval-augmented-generation
- agents
- generative-ai-tools
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
- workflow-schedule
- azkaban
- airflow
- task-scheduler
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
The official Open-Asset-Importer-Library Repository. Loads 40+ 3D-file-formats into one unified and clean data structure.
- asset-pipeline
- assets
- game-development
- c-plus-plus
Proxy based Redis cluster solution supporting pipeline and scaling dynamically
Always know what to expect from your data.
- pipeline-tests
- dataquality
- datacleaning
- datacleaner
Kubescape is an open-source Kubernetes security platform for your IDE, CI/CD pipelines, and clusters. It includes risk analysis, security, compliance, and misconfiguration scanning, saving Kubernetes users and administrators precious time, effort, and resources.
- kubernetes
- security
- nsa
- mitre-attack
Spug is a lightweight agent-free automatic operation and maintenance platform designed for small and medium-sized enterprises. It integrates host management, host batch execution, host online terminal, file management, application release and deployment, pipelines, online task planning, configuration center, monitoring, alarm and so on.
- ops
- operations
- devops
- webssh
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
StreamDiffusion: A Pipeline-Level Solution for Real-Time Interactive Generation
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
🌌 A complete search engine and RAG pipeline in your browser, server or edge network with support for full-text, vector, and hybrid search in less than 2kb.
- data-structures
- full-text
- search
- typo-tolerance
httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
AlphaFold 3 inference pipeline.