Repository Index

Learn how to develop, deploy and iterate on production-grade ML applications.

• machine-learning
• deep-learning
• pytorch
• natural-language-processing

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

• data-engineering
• kafka
• spark
• dbt

This is a repo with links to everything you'd ever want to learn about data engineering

• apachespark
• awesome
• bigdata
• data

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

• h2o
• machine-learning
• data-science
• deep-learning

The Hunting ELK

• hunting
• elasticsearch
• kibana
• logstash

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering

Easy pipelines for pandas DataFrames.

• pandas
• pandas-dataframe
• pipeline
• data

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.

Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink, ...

• benchmarks
• low-latency
• streaming

Docker container for Kafka - Spark Streaming - Cassandra

Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.

• big-data
• bigdata
• spark
• spark-sql

Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.

• notebooks
• pyspark-notebook
• juypter-notebook
• spark

GTFS Data Pipeline for TfNSW Bus Datasets

• gtfs
• gtfs-static
• gtfs-realtime
• tfnsw