Repository Index

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

• pandas-profiling
• pandas-dataframe
• statistics
• jupyter-notebook

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

• machine-learning
• ml
• model-validation
• data-validation

AI Infra / AI Orchestration / AI Control Plane

• deep-learning
• machine-learning
• artificial-intelligence
• data-science

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering

Elyra extends JupyterLab with an AI centric approach.

• notebooks
• notebook-jupyter
• jupyterlab
• jupyterlab-extensions

Jupyter magics and kernels for working with remote Spark clusters

• spark
• kernel
• cluster
• livy

📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)

• analytics
• bi-server
• java
• big-data

Easy pipelines for pandas DataFrames.

• pandas
• pandas-dataframe
• pipeline
• data

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.

Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink, ...

• benchmarks
• low-latency
• streaming

Docker container for Kafka - Spark Streaming - Cassandra

Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.

• big-data
• bigdata
• spark
• spark-sql

Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.

• notebooks
• pyspark-notebook
• juypter-notebook
• spark

GTFS Data Pipeline for TfNSW Bus Datasets

• gtfs
• gtfs-static
• gtfs-realtime
• tfnsw