Repository Index

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

• data-engineering
• kafka
• spark
• dbt

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering

Turns Data and AI algorithms into production-ready web applications in no time.

• automation
• data-engineering
• data-ops
• data-visualization

Workflow Engine for Kubernetes

• workflow
• kubernetes
• argo
• dag

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science

Roadmap to becoming a data engineer in 2021

• data-engineer-roadmap
• data-engineering
• cloud
• roadmap

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

• xonsh
• devops
• iterm2
• data-engineering

Open Source Feature Flags, Experimentation, and Product Analytics

• abtesting
• statistics
• abtest
• experimentation

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

• aws
• gcp
• azure
• sql

lakeFS - Data version control for your data lake | Git for data

• data-engineering
• data-versioning
• go
• object-storage

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

• data-engineering
• data-integration
• data-pipeline
• data-quality

Free and open source schema versioning and database migration made natively with .NET/6. NEW THIS MAY 2022! v1.3.15 released!

• sql
• yuniql
• database-migrations
• sqlserver

Realistic synthetic events for testing, demos, and pipelines — streamed live or generated in bulk

• clickhouse
• data-engineering
• data-generation
• devops

A lightweight Snowflake emulator built with Go and DuckDB for local development and testing

• cicd
• database
• docker
• duckdb

ETL scripts for EOS.

• eos
• eosio
• etl
• apache-beam

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.

• collaborative-editing
• conflict-resolution
• crdt
• data-engineering