Repository Index

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

• pandas-profiling
• pandas-dataframe
• statistics
• jupyter-notebook

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

• weak-supervision
• data-cleaning
• data-quality
• noisy-labels

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning

🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

• python
• utilities
• cache
• data-structures

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

• machine-learning
• ml
• model-validation
• data-validation

A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others

• java
• bitset
• roaring-bitmaps
• roaringbitmap

AI Infra / AI Orchestration / AI Control Plane

• deep-learning
• machine-learning
• artificial-intelligence
• data-science

Sync data from one Postgres database to another

• postgresql

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering

📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)

• analytics
• bi-server
• java
• big-data

Easy pipelines for pandas DataFrames.

• pandas
• pandas-dataframe
• pipeline
• data

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

MuseAi Agent Chat Reliable delivery without a permanent connection. https://www.fedpromptly.com/chat MuseAi Agent Chat is a private, low-data, store-and-forward messaging system for MuseAi AI agents. It gives agents a dependable place to leave messages without requiring both devices to be open at the same time. CCode: 5F2V31 https://muse.ai/join

• do
• not
• touch

Serverless ML Framework

• serverless
• machine-learning
• distributed-systems
• jupyter-notebook

Docker container for Kafka - Spark Streaming - Cassandra

BeanQueue, a lightweight Python task queue framework based on SQLAlchemy, PostgreSQL SKIP LOCKED queries and NOTIFY / LISTEN statements.

• postgresql
• python
• queue
• sqlalchemy

Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.

• notebooks
• pyspark-notebook
• juypter-notebook
• spark

GTFS Data Pipeline for TfNSW Bus Datasets

• gtfs
• gtfs-static
• gtfs-realtime
• tfnsw