SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Repository Index
47,133 repositories indexed
1–22 of 22 · Most stars
Logstash - transport and process your logs, events, or other data
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
The leader in Customer Data Infrastructure
- snowplow
- analytics
- data
- data-pipeline
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Python Streaming DataFrames for Kafka
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
Polo travels through your database and creates sample snapshots so you can work with real world data in development.
- database
- snapshot
- productivity
- activerecord
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!
- code-generator
- etl
- dotnet-core
- dotnet
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
Graph-based security analysis for everyone
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
Database synchronizer for Magento 2 (and Wordpress), based on Magerun2. Keeping development, staging and production in sync easily.
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
Reads EVE files into SQL as well as search stored data.