Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 33 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
All-in-one IoT Platform - Device management, data collection, processing and visualization.
- visualization
- iot
- iot-platform
- mqtt
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
- pandas-profiling
- pandas-dataframe
- statistics
- jupyter-notebook
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
The leader in Customer Data Infrastructure
- snowplow
- analytics
- data
- data-pipeline
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
- cloud-native
- streaming
- rust
- real-time
High-performance time-series aggregation for PostgreSQL
- realtime
- stream-processing
- analytics
- sql
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
Lightweight data stream processing engine for IoT edge
Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster
Data API Framework for AI Agents and Data Apps
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.
oneAPI Data Analytics Library (oneDAL)
- onedal
- data-analysis
- oneapi
- machine-learning
Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.
- data-analytics
- data-cleaning
- data-cleansing
- data-engineering
Haskell on Apache Spark.
- apache-spark
- spark
- haskell
- analytics
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
An effort to bring Google's Firebase C++ API to Qt + QML