The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
- claude-code
- codex
- data-processing
- harness-engineering
Repository Index
76,855 visible repositories in the index
25–48 of 54 · Most stars
Self-hosted platform to unify wearable health data through one AI-ready API.
- wearables
- apple-health
- apple-watch
- fitness
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
- unstructured-data
- low-code
- developer-tools
- etl
Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.
- dataprep
- data-science
- datapreparation
- dataconnector
Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.
Python CLI utility and library for manipulating SQLite databases
- sqlite
- python
- datasette
- sqlite-database
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Scalable data pre processing and curation toolkit for LLMs
- data-curation
- llm
- data
- data-prep
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
Python Streaming DataFrames for Kafka
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
The data transformation tool the open-source community deserves.
- data-engineering
- duckdb
- data-lineage
- data-pipeline
A decision-model-based ontology & knowledge-graph builder — builds governed knowledge graphs from your documents, every fact gated and traceable to its evidence.
- graphrag
- information-extraction
- knowledge-graph
- llm
High performance Kafka consumer for InfluxDB. Supports collectd message formats.
- kafka-influxdb
- kafka-consumer
- logstash
- influxdb
Event-driven OneLake pipeline with automated Upsert merge and Python reporting
- activator
- data-engineering
- data-factory
- data-pipeline
Doc manager for Neo4j
Code to download, process, and analyse the Online Encyclopedia of Integer Sequences
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
A simple data project for Steam data
- steam
- steam-data
- steam-api
- python
Interfaces ham radio APRS-IS servers and saves packet data into an influxdb database
Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.
- postgres
- postgresql
- python
- infectious-disease