Repository Index

lakeFS - Data version control for your data lake | Git for data

• data-engineering
• data-versioning
• go
• object-storage

Utils for streaming large files (S3, HDFS, gzip, bz2...)

• python
• s3
• hdfs
• webhdfs

Compositional, streaming I/O library for Scala

• scala
• stream
• streaming-computation
• fp

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe

Canonical GTFS Validator project for schedule (static) files.

• gtfs-dataset
• gtfs-validator
• gtfs-zip
• gtfs-feed

Event-Sourcing and Domain-Driven Design with F# and EventStore

:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

• aws
• textract
• lambda
• dynamodb

Real-time P2P messenger using go-ipfs pubsub. TUI. End-to-end encrypted texting & file-sharing. NAT traversal.

• p2p
• tui
• ipfs
• pubsub

Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.

• big-data
• bigdata
• spark
• spark-sql

GTFS Data Pipeline for TfNSW Bus Datasets

• gtfs
• gtfs-static
• gtfs-realtime
• tfnsw