Repository Index

MLeap: Deploy ML Pipelines to Production

• scikit-learn
• spark
• data-pipelines
• transformers

visual data prep powered by python

• data
• data-pipelines
• etl
• structured-data

zerocode-tdd is a community-developed, free, open-source, outcome-driven automated testing framework for Data Pipelines, ETL, REST API, Kafka(Data Streams), Databases and Load scenarios. all defined in simple JSON or YAML — with zero coding.

• soap
• assertions
• java
• api

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

• change-data-capture
• data
• data-collection
• data-engineering

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

• hpc
• workflow
• workflow-automation
• workflow-management

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

• go
• data-integration
• etl
• data-pipeline

Streaming reactive and dataflow graphs in Python

• python
• python3
• stream
• data-pipeline

:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

• aws
• textract
• lambda
• dynamodb

Build and deploy a serverless data pipeline on AWS with no effort.

• aws-cdk
• data-pipeline
• stepfunctions
• glue

Ordered-concurrently a library for concurrent processing with ordered output in Go. Process work concurrently and returns output in a channel in the order of input. It is useful in concurrently processing items in a queue, and get output in the order provided by the queue.

• concurrent
• golang
• golang-library
• data-science

Reverse ETL for the code-first data stack

• bigquery
• cli
• data-engineering
• dbt

GTFS Data Pipeline for TfNSW Bus Datasets

• gtfs
• gtfs-static
• gtfs-realtime
• tfnsw