The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
- system-design
- backend
- scalability
- interview
Repository Index
47,133 repositories indexed
1–24 of 49 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Learn how to develop, deploy and iterate on production-grade ML applications.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Cross-platform, customizable ML solutions for live and streaming media.
- mediapipe
- c-plus-plus
- computer-vision
- deep-learning
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
- applied-machine-learning
- production
- applied-data-science
- machine-learning
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
- machine-learning
- ai
- ml
- mlflow
A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.
- notebooks
- python
- data-science
- machine-learning
☁️ Build multimodal AI applications with cloud-native stack
- neural-search
- cloud-native
- deep-learning
- machine-learning
An open source cybersecurity protocol for syncing decentralized graph data.
Workflow Engine for Kubernetes
- workflow
- kubernetes
- argo
- dag
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
- machine-learning
- decision-trees
- gradient-boosting
- gbm
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
- dataframe
- python
- bigdata
- tabular-data
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
The Open Source Feature Store for AI/ML
- machine-learning
- features
- ml
- big-data
🌊 Online machine learning in Python
- incremental-learning
- machine-learning
- python
- online-learning
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
Machine learning platform for Web developers
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.