Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Repository Index
47,133 repositories indexed
1–24 of 46 · Most stars
Logstash - transport and process your logs, events, or other data
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
- pandas-profiling
- pandas-dataframe
- statistics
- jupyter-notebook
Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.
Efficient reliable UDP unicast, UDP multicast, and IPC message transport
- messaging
- java
- c-plus-plus
- ipc
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.
- machine-learning
- ml
- model-validation
- data-validation
A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases. The VerneMQ mission is active & the project maintained. Thank you for your support!
- mqtt
- m2m
- scalable
- broker
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
- workflow
- governance
- azkaban
- davinci
Database Subsetting and Relational Data Browsing Tool.
Import CSV and JSON into PostgreSQL the easy way
HiveMQ CE is a Java-based open source MQTT broker that fully supports MQTT 3.x and MQTT 5. It is the foundation of the HiveMQ Enterprise Connectivity and Messaging Platform
A lightweight, high-performance go mqtt server(v3.0|v3.1.1|v5.0) supporting distributed cluster
Healthcare data infrastructure for next generation care delivery.
- api
- healthcare
- open-source
- soc2
Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
A simple mediator for .Net for sending command, publishing event and request response with pipelines supported
- mediator
- bus
- cqrs
- ddd
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰
- csv-import
- data-import
- data-mapping
- data-review
Silverback is a message bus and broker integration library for .NET. It is designed to simplify event-driven architectures and asynchronous messaging, with first-class support for Apache Kafka and MQTT.
- microservice
- messaging
- kafka
- silverback
Modular .NET messaging for commands, queries, events, durable inbox and outbox processing, sagas, and broker transports.
- cqrs
- cqs
- csharp
- dotnet-core
High performance Kafka consumer for InfluxDB. Supports collectd message formats.
- kafka-influxdb
- kafka-consumer
- logstash
- influxdb
Data Quality and Observability platform for the whole data lifecycle, from profiling new data sources to full automation with Data Observability. Configure data quality checks from the UI or in YAML files, let DQOps run the data quality checks daily to detect data quality issues.
- data-ops
- data-quality
- data-quality-checks
- data-quality-measurement