Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–21 of 21 · Most stars
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
A light-weight, flexible, and expressive statistical data testing library
- pandas
- validation
- schema
- dataframes
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
WebSocket and WAMP in Python for Twisted and asyncio
- python
- websocket
- real-time
- wamp
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
Crossbar.io - WAMP application router
MLBox is a powerful Automated Machine Learning python library.
- machine-learning
- auto-ml
- kaggle
- deep-learning
Jupyter magics and kernels for working with remote Spark clusters
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
The world's cleanest AutoML library ✨ - Do hyperparameter tuning with the right pipeline abstractions to write clean deep learning production pipelines. Let your pipeline steps have hyperparameter spaces. Design steps in your pipeline like components. Compatible with Scikit-Learn, TensorFlow, and most other libraries, frameworks and MLOps environments.
- pipeline
- pipeline-framework
- machine-learning
- deep-learning
The repository host gives no description.
- hacktoberfest
- data
- datalineage
- dbt
Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
- apache-iceberg
- apache-spark
- data-engineering
- delta-lake
Read and write Neuroglancer datasets programmatically.
- neuroglancer
- volumetric-data
- python
- cloud
Realistic synthetic events for testing, demos, and pipelines — streamed live or generated in bulk
- clickhouse
- data-engineering
- data-generation
- devops
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
BeanQueue, a lightweight Python task queue framework based on SQLAlchemy, PostgreSQL SKIP LOCKED queries and NOTIFY / LISTEN statements.
Simple MQTT Client.
- mqtt
- client
- mqtt-client
- python
A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.
- anomaly-detection
- bigquery
- cli
- data-engineering
ETL scripts for EOS.
- eos
- eosio
- etl
- apache-beam
Pipeline de ingestão e busca semântica de documentos para RAG — com rate limiting, circuit breaker e fila resiliente.