:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Repository Index
111,821 visible repositories in the index
97–120 of 215 · Most stars
Concurrent Python made simple
- asyncio
- concurrency
- python
- threading
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
Elasticsearch File System Crawler (FS Crawler)
- java
- elasticsearch
- crawler
- tika
Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.
- hadoop
- hive
- database
- clickhouse
Support agile Ontology DataOps Based on Flink, DataX and Flink-CDC with Web-UI
Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync
- elasticsearch
- elasticsearch-sync
- postgresql
- change-data-capture
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.
- blockchain
- blockchain-indexer
- ethereum
- etl
Import CSV and JSON into PostgreSQL the easy way
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
- elasticsearch
- go
- golang
- mongodb
Python Extract Transform and Load Tables of Data
Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
- apache-beam
- dataflow-templates
- google-cloud-dataflow
- google-cloud-storage
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
- integration
- reactive
- akka
- akka-stream
Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)
- search
- search-engine
- search-interface
- ocr
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
- logstash
- elk
- kafka
- elasticsearch
The swiss army knife of healthcare integration.
- mirth
- mirth-connect
- hl7
- healthcare
Python Fast Dataflow programming framework for Data pipeline work( Web Crawler,Machine Learning,Quantitative Trading.etc)
DataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步,分布式、可扩展的数据交换平台。
- data-replication
- datalink
- data-exchange
- realtime-replication
Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.
Scratch is a swiss army knife for big data.
- clickhouse
- data-warehouse
- hacktoberfest
- olap
使用 golang 模仿的 Logstash。用于消费 Kafka 数据,处理后写入 ES、Clickhouse 等。
- kafka
- elasticsearch
- logstash
- golang