Sign in

Repository Index

47,133 repositories indexed

Tag: Data pipelines and messaging
Tag
Filters

Filters

Clear all filters

1–18 of 18 · Most stars

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  • python
  • luigi
  • orchestration-framework
  • scheduling

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

  • metadata
  • datadiscovery
  • dataquality
  • data-profiling

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

  • data
  • ingestion
  • replication
  • management

Distributed, Scalable, Fault-Tolerant, Minimalistic Workflow Engine - No DAGs, No YAML, No Cumbersome Diagrams, Just Code

  • business-process
  • kafka
  • kafka-streams
  • micorservices

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

  • etl
  • etl-framework
  • etl-pipeline
  • python