Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Repository Index
47,133 repositories indexed
1–19 of 19 · Most stars
Apache Hadoop
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
- workflow
- governance
- azkaban
- davinci
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Run MapReduce jobs on Hadoop or Amazon Web Services
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
RabbitMQ wire tap and swiss army knife
- rabbitmq
- tap
- wiretap
- golang
Distributed Temporal Graph Analytics with Apache Flink
- graph
- property-graph
- distributed-graph-analytics
- apache-flink
Python MapReduce library written in Cython. Visit us in #hadoopy on freenode. See the link below for documentation and tutorials.
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
Blockchain Indexing Engine
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Partition (W)ARC Files by MIME Type and Year
- web-archiving
- webarchive
- hadoop
- warc