Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
132,241 visible repositories in the index
25–48 of 65 · Most stars
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes
Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 437 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.
TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
Scratch is a swiss army knife for big data.
使用 golang 模仿的 Logstash。用于消费 Kafka 数据,处理后写入 ES、Clickhouse 等。
Intake is a lightweight package for finding, investigating, loading and disseminating data.
MetaForge is integrated software for data transformation, integration, and visualization. Support a variety of data sources, file types, big data related database, relational database, NoSQL database, etc. Through the software can realize the management of multiple data sources, the data under the source of various operations conversion, making dat
Data syncing in golang for ClickHouse.
The MongoDB Spark Connector
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
🌱 The ultimate solution for populating your MongoDB database.
Graph-based security analysis for everyone
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
The data transformation tool the open-source community deserves.
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰
Blockchain Indexing Engine