Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Repository Index
47,133 repositories indexed
1–16 of 16 · Most stars
Apache Hadoop
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
- workflow
- governance
- azkaban
- davinci
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Run MapReduce jobs on Hadoop or Amazon Web Services
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Apache Calcite Go
- geospatial
- calcite
- java
- big-data
Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.
A production-grade HBase ORM library that makes accessing HBase clean, fast and fun (Can also be used as Bigtable ORM)
- hbase
- orm
- mapreduce
- object-mapping
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Partition (W)ARC Files by MIME Type and Year
- web-archiving
- webarchive
- hadoop
- warc