Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 99 · Most stars
Learn how to develop, deploy and iterate on production-grade ML applications.
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Cross-platform, customizable ML solutions for live and streaming media.
- mediapipe
- c-plus-plus
- computer-vision
- deep-learning
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
- applied-machine-learning
- production
- applied-data-science
- machine-learning
A high-performance observability data pipeline.
- logs
- metrics
- observability
- forwarder
All-in-one IoT Platform - Device management, data collection, processing and visualization.
- visualization
- iot
- iot-platform
- mqtt
Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
- telegraf
- monitoring
- time-series
- metrics
QuestDB is a high performance, open-source, time-series database
- time-series
- low-latency
- database
- sql
Workflow Engine for Kubernetes
- workflow
- kubernetes
- argo
- dag
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
A curated list of awesome big data frameworks, ressources and other awesomeness.
- awesome
- bigdata
- data-warehouse
- data-analytics
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
- kafka
- cpp
- kubernetes
- go
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
- relational-database
- message-broker
- hadoop-ecosystem
- nosql
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
Building event-driven applications the easy way in Go.
- watermill
- event-driven
- event-sourcing
- sagas
BullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL
Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Python Stream Processing
- kafka-streams
- kafka
- python
- asyncio
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data