Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Repository Index
47,133 repositories indexed
1–24 of 26 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
A network event stream processing system, in Clojure.
- riemann
- events
- clojure
- monitoring
Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.
- machine-learning
- ml
- model-validation
- data-validation
Micro second messaging that stores everything to disk
- chronicle
- queue
- java
- performance
Ruby and Rails efficient Kafka processing framework
- karafka-framework
- kafka
- kafka-client
- kafka-clients
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
- change-data-capture
- data
- data-collection
- data-engineering
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
Low-code tool for automating actions on real time data | Stream processing for the users.
- flink
- flink-kafka
- apache-flink
- gui
Data validation toolkit for assessing and monitoring data quality.
- data-validation
- tabular-data
- data-testing
- data-quality
Canonical GTFS Validator project for schedule (static) files.
- gtfs-dataset
- gtfs-validator
- gtfs-zip
- gtfs-feed
Automatic data change tracking for PostgreSQL
- audit-log
- audit-trail
- change-data-capture
- change-tracking
Modular, Production-Ready, Open-Source EHR
- ehr
- electronic-health-record
- fhir
- health
A CLI tool for Kafka
StreamFlow™ is a stream processing tool designed to help build and monitor processing workflows.
📜 A CLI toolkit for extracting and working with your digital history
- chronicle
- chronicle-etl
- personal-data
- etl
Daggy - Data Aggregation Utility and C/C++ developer library for data streams catching
- serverless-framework
- streaming
- stream-processing
- qt
Type-safe, persistent, embedded task queues and background job runner w/ SQLite. Web monitoring UI included.
- go
- golang
- job-queue
- job-scheduler
Scalable Full History API Solution for Antelope (former EOSIO) based blockchains
- antelope
- api
- big-data
- blockchain
A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.
- anomaly-detection
- bigquery
- cli
- data-engineering