Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Repository Index
47,133 repositories indexed
1–14 of 14 · Most stars
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
PySpark test helper methods with beautiful error messages
Essential Spark extensions and helper methods ✨😲
- spark
- dataframe
Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)
StreamFlow™ is a stream processing tool designed to help build and monitor processing workflows.
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.
- python3
- data-science
- machine-learning
- python
A set of open-source .NET libraries helping with DDD (domain-driven design) and CQRS (command query responsibility segregation).
- ddd
- cqrs
- nhibernate
- asp-net-core
Helpers & syntactic sugar for PySpark.
An open source event management platform focused on Hackathons
- event
- event-management-platform
- hackathon
- hackathons
Generic Task behavior that helps encapsulate errors and recover from them in classic GenStage workers.
- elixir-lang
- elixir
- hex
- package