Sign in

Repository Index

97,836 visible repositories in the index

Tags: Data pipelines and messaging
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 127 · Most stars

SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.

  • distributed-storage
  • s3
  • hdfs
  • fuse

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  • python
  • luigi
  • orchestration-framework
  • scheduling

SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

  • metadata
  • datadiscovery
  • dataquality
  • data-profiling

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

  • dbt-viewpoint
  • slack
  • pypa
  • data-modeling

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

  • kedro
  • hacktoberfest
  • mlops
  • python

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

  • data-integration
  • high-performance
  • offline
  • real-time

Apache KIE (Knowledge Is Everything) is the home of Drools, OptaPlanner, jBPM, and Kogito.

  • rule-engine
  • java
  • dmn
  • hacktoberfest

Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.