Open Data Platform for analysts, quants and AI agents.
- python
- machine-learning
- finance
- stocks
Repository Index
76,356 visible repositories in the index
1–24 of 173 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Run frontier AI locally.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Apache Flink
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
- python
- workflow
- data-engineering
- data-science
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
Graph-Native Infrastructure for Context and Accountable AI Systems
- ai
- ai-governance
- artificial-intelligence
- context-engineering
Parallel computing with task scheduling
- dask
- python
- pydata
- numpy
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
- pandas-profiling
- pandas-dataframe
- statistics
- jupyter-notebook
Node-based Visual Programming Toolbox
- photogrammetry
- computer-vision
- 3d-reconstruction
- alicevision
Always know what to expect from your data.
- pipeline-tests
- dataquality
- datacleaning
- datacleaner
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
- weak-supervision
- data-cleaning
- data-quality
- noisy-labels
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.
- rust
- database
- serverless
- bigdata
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam