High performance Kafka consumer for InfluxDB. Supports collectd message formats.
- kafka-influxdb
- kafka-consumer
- logstash
- influxdb
Repository Index
47,133 repositories indexed
73–95 of 95 · Most stars
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.
- python3
- data-science
- machine-learning
- python
Read and write Neuroglancer datasets programmatically.
- neuroglancer
- volumetric-data
- python
- cloud
Free Course : Data, Machine Learning, Deep Learning and AI Engineering, the long way round: the math, the code, the systems and the security. 173 lessons, 166 runnable labs, no shortcuts. Free under CC BY-NC 4.0.
- ai-engineering
- course
- data-engineering
- deep-learning
ENIGMA HALFpipe is a user-friendly software that facilitates reproducible analysis of fMRI data
- neuroimaging
- reproducibility
- pipeline
- meta-analysis
Build and deploy a serverless data pipeline on AWS with no effort.
- aws-cdk
- data-pipeline
- stepfunctions
- glue
Realistic synthetic events for testing, demos, and pipelines — streamed live or generated in bulk
- clickhouse
- data-engineering
- data-generation
- devops
Helpers & syntactic sugar for PySpark.
A micro workers class that easily manages a pool of threads to optimise simultaneous jobs and jobs endings, for MicroPython (used on Pycom modules & ESP32)
- pycom
- micropython
- lopy
- wipy
Stream Processing Made Easy
- machine-learning
- reactive-programming
- kafka
- reactive-systems
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
BeanQueue, a lightweight Python task queue framework based on SQLAlchemy, PostgreSQL SKIP LOCKED queries and NOTIFY / LISTEN statements.
Async MQTT client for CPython and MicroPython with asyncio support.
- async-python
- asyncio
- embedded
- embedded-systems
Declarative data quality engine. Define checks in YAML, run anywhere.
- airflow
- anomaly-detection
- data-contracts
- data-quality
Simple MQTT Client.
- mqtt
- client
- mqtt-client
- python
A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.
- anomaly-detection
- bigquery
- cli
- data-engineering
🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines
- etl
- etl-framework
- etl-pipeline
- python
Blazingly fast data comparison tool for Python, powered by Rust. Compare massive CSV/Parquet datasets instantly.
ETL scripts for EOS.
- eos
- eosio
- etl
- apache-beam
Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.
- collaborative-editing
- conflict-resolution
- crdt
- data-engineering
Open, audit-grade agentic data quality framework with portable industry packs
Data ingestion engine — Singer taps to Parquet in cloud buckets
Pipeline de ingestão e busca semântica de documentos para RAG — com rate limiting, circuit breaker e fila resiliente.