This is a repo with links to everything you'd ever want to learn about data engineering
Repository Index
47,133 repositories indexed
1–23 of 23 · Most stars
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
The official home of the Presto distributed SQL query engine for big data
A curated list of awesome big data frameworks, ressources and other awesomeness.
PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
The leader in Customer Data Infrastructure
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
Distributed stream processing engine in Rust
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Data processing & ETL framework for Ruby
Nessie: Transactional Catalog for Data Lakes with Git-like semantics
Clean APIs for data cleaning. Python implementation of R package Janitor
visual data prep powered by python
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
Easy pipelines for pandas DataFrames.
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
No description provided by the repository host.
BENERATOR is a leading software solution to generate, obfuscate, pseudonymize and migrate data for development, testing, and training purposes with a model-driven approach.
A curated list of awesome Qlik extensions and resources for Qlik Sense and QlikView
Data ingestion engine — Singer taps to Parquet in cloud buckets