Sign in

Repository Index

132,241 visible repositories in the index

Tags: Data pipelines and messaging
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 215 · Most stars

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  • python
  • luigi
  • orchestration-framework
  • scheduling

SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

  • workflow-schedule
  • azkaban
  • airflow
  • task-scheduler

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

  • kedro
  • hacktoberfest
  • mlops
  • python

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

  • data-integration
  • high-performance
  • offline
  • real-time

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

  • data
  • data-cleaning
  • data-pipelines
  • data-processing