Utils for streaming large files (S3, HDFS, gzip, bz2...)
- python
- s3
- hdfs
- webhdfs
89,572 visible repositories in the index
1–5 of 5 · Most stars
Utils for streaming large files (S3, HDFS, gzip, bz2...)
Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.
Streaming reactive and dataflow graphs in Python
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz