pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Repository Index
47,133 repositories indexed
25–48 of 87 · Most stars
Blazing-fast, Neuro-symbolic Data-Wrangling toolkit
- csv
- data-wrangling
- opendata
- data-engineering
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
Apache DataFusion Ballista Distributed Query Engine
- arrow
- big-data
- dataframe
- olap
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Lightweight data stream processing engine for IoT edge
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Stream Processing and Complex Event Processing Engine
- streaming-data-processing
- cep
- stream-processing
- complex-event-processing
Python Streaming DataFrames for Kafka
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
Clean APIs for data cleaning. Python implementation of R package Janitor
- pandas
- dataframe
- data
- cleaning-data
Distributed Stream Processing
- stream-processing
- stream-processor
- stream-processing-engine
- pony-language
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
Hopsworks - Data-Intensive AI platform with a Feature Store
- feature-store
- aws
- azure
- data-science
A RocksDB compliant high performance scalable embedded key-value store
- key-value-store
- kvs
- storage-engine
- speedb
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
- change-data-capture
- data
- data-collection
- data-engineering
Polyglot workflows without leaving the comfort of your technology stack.
- orchestration-framework
- scheduling
- workflow
- event-driven
Workflow Automation Engine
- workflow-engine
- workflow
- cross-platform
- automation
Tenzir is the data pipeline engine for security teams.
- incident-response
- threathunting
- siem
- soc
Lightweight real-time big data streaming engine over Apache Pekko
- akka
- stream-processing
- bigdata
- scala
Low-code tool for automating actions on real time data | Stream processing for the users.
- flink
- flink-kafka
- apache-flink
- gui