Sign in

Repository Index

142,273 visible repositories in the index

Tags: Data pipelines and messaging
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 143 · Most stars

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  • python
  • luigi
  • orchestration-framework
  • scheduling

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

  • workflow-schedule
  • azkaban
  • airflow
  • task-scheduler

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

  • kedro
  • hacktoberfest
  • mlops
  • python

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

  • data
  • data-cleaning
  • data-pipelines
  • data-processing

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.