Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
Repository Index
47,133 repositories indexed
73–96 of 225 · Most stars
Machine learning platform for Web developers
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
A curated list of awesome OpenGL libraries, debuggers and resources.
A LiDAR odometry pipeline that just works
- 3d-mapping
- icp
- lidar-slam
- slam
HTML processing filters and utilities
Translated【A common used C++ & Python DAG framework】 A general-purpose, third-party-dependency-free, cross-platform parallel computing framework based on flow graphs, listed in awesome-cpp. Welcome to star & fork & discuss
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
A lightweight stream processing library for Go
CLI task management & automation tool
- python
- build-tool
- build-automation
- task-runner
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
Elyra extends JupyterLab with an AI centric approach.
- notebooks
- notebook-jupyter
- jupyterlab
- jupyterlab-extensions
A full spaCy pipeline and models for scientific/biomedical documents.
- scientific-documents
- spacy
- custom-pipes
- nlp
Azure Pipelines Agent 🚀
Composable concurrency toolkit for Go
- concurrency
- channels
- go
- golang
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Twitter Image Pipeline is a robust and performant image loading and caching framework for iOS clients
- image-pipeline
- ios
- swift
- objective-c
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
A categorized collection of FFmpeg commands for video automation pipelines.
- automation
- cheatsheet
- ffmpeg
- ffprobe
A deep learning toolkit for healthcare AI: EHR (MIMIC, eICU, OMOP), clinical text, signals and imaging. 30+ models, reproducible pipelines.
- healthcare
- data-mining
- deep-learning
- preprocessing
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Cookbook for Node on FHIR and the clinical Meteor.js build pipeline. Create FDA, HIPPA, and HL7 compliant Javascript/Node applications!
Python Streaming DataFrames for Kafka
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality