Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Repository Index
47,133 repositories indexed
1–24 of 49 · Most stars
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
- steampipe
- postgresql
- postgresql-fdw
- cloud
Migrate to PostgreSQL in a single command!
- postgresql
- migration
- mysql
- mssql
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
AI Infra / AI Orchestration / AI Control Plane
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Database Subsetting and Relational Data Browsing Tool.
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
High-performance time-series aggregation for PostgreSQL
- realtime
- stream-processing
- analytics
- sql
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.
- dataprep
- data-science
- datapreparation
- dataconnector
Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Lightweight and extensible compatibility layer between dataframe libraries!
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Microservice native message and event store for Postgres
- event-store
- event-sourcing
- event-driven
- pub-sub
Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality