Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Repository Index
47,133 repositories indexed
1–20 of 20 · Most stars
Apache Hadoop
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
- workflow
- governance
- azkaban
- davinci
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Run MapReduce jobs on Hadoop or Amazon Web Services
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
Complete working example of using Domain Driven Design (DDD), Hexagonal Architecture, CQRS, Event Sourcing (ES), Event Driven Architecture (EDA), Behaviour Driven Development (BDD) using TypeScript and NestJS. Like what you see? Don't forget to star! ⭐ ^^^
PHP 7.4 EventStore Implementation
- prooph
- php
- event-store
- event-sourcing
The GIS Tools for Hadoop are a collection of GIS tools for spatial analysis of big data.
- spatial-analysis
- hadoop
Distributed Temporal Graph Analytics with Apache Flink
- graph
- property-graph
- distributed-graph-analytics
- apache-flink
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Apache Calcite Go
- geospatial
- calcite
- java
- big-data
Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.
A production-grade HBase ORM library that makes accessing HBase clean, fast and fun (Can also be used as Bigtable ORM)
- hbase
- orm
- mapreduce
- object-mapping
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Partition (W)ARC Files by MIME Type and Year
- web-archiving
- webarchive
- hadoop
- warc