Repository Index

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

• metadata
• datadiscovery
• dataquality
• data-profiling

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

• bloom
• deep-learning
• distributed-systems
• language-models

Build, Manage and Deploy AI/ML Systems

• machine-learning
• model-management
• ai
• ml

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

• mlops
• machine-learning
• data-science
• production-ready

Blazing-fast, Neuro-symbolic Data-Wrangling toolkit

• csv
• data-wrangling
• opendata
• data-engineering

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

• datalake
• lakehouse
• metadata
• federated-query

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering

A high available,high performance distributed messaging system.

Data structure and algorithm library for go, designed to provide functions similar to C++ STL

• list
• vector
• deque
• queue

Lightweight and Extensible Infrastructure for Building Web Applications - Web Application Framework

• dotnet-core
• efcore
• aspnet-core
• solid

Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.

RabbitMQ wire tap and swiss army knife

• rabbitmq
• tap
• wiretap
• golang

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

Web application for distributed compute analysis of Archive-It web archive collections.