Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
67,262 visible repositories in the index
73–96 of 993 · Most stars
Chat with your documents on your local device using GPT models. No data leaves your device and 100% private.
🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools
Best Practices on Recommendation Systems
- machine-learning
- recommender
- ranking
- deep-learning
OCR, layout analysis, reading order, table recognition in 90+ languages
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.
- forecasting
- r
- python
open-source agentic AI data assistant for the next generation of AI + Data products.
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
- ai
- cpp
- data-pipeline
- data-processing
Toolkit for linearizing PDFs for LLM datasets/training
Faker is a Python package that generates fake data for you.
Turns Data and AI algorithms into production-ready web applications in no time.
- automation
- data-engineering
- data-ops
- data-visualization
SQL databases in Python, designed for simplicity, compatibility, and robustness.
- python
- sql
- sqlalchemy
- pydantic
GenBI (Generative BI) for AI agents, an open-source, governed text-to-SQL through an open context layer that turns natural-language questions into trusted dashboards, charts, and SQL across 20+ data sources, such as BigQuery, Snowflake, PostgreSQL, ClickHouse, Amazon Redshift, Databricks and more.
- bigquery
- duckdb
- llm
- postgresql
Datasets, Transforms and Models specific to Computer Vision
🪄 Data Formulator is an interactive AI-powered data analysis system makes it easy to connect, explore and visualize data.
Memori is agent-native memory infrastructure. A LLM-agnostic layer that turns agent execution and conversation into structured, persistent state for production systems. Built for enterprise, Memori works with the data infrastructure you already run, no rip-and-replace, and deploys across managed cloud, single-tenant cloud, VPC, and on-premises.
CVAT is a leading data annotation platform for image, video, audio, and 3D datasets. It offers open-source, cloud, and enterprise products, as well as labeling services, with AI-assisted labeling, quality assurance, team collaboration, analytics, and developer APIs.
- video-annotation
- computer-vision
- computer-vision-annotation
- image-annotation
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
- deepseek-v4
- edge
- glm
- inference
Official repository for IPython itself. Other repos in the IPython organization contain things like the website, documentation builds, etc.
- ipython
- jupyter
- data-science
- python
pix2tex: Using a ViT to convert images of equations into LaTeX code.
- machine-learning
- transformer
- im2latex
- deep-learning
Topic Modelling for Humans
- gensim
- topic-modeling
- information-retrieval
- machine-learning
Avatars for Zoom, Skype and other video-conferencing apps.
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science