:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Repository Index
55,640 visible repositories in the index
49–72 of 116 · Most stars
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
- change-data-capture
- data
- data-collection
- data-engineering
Healthcare data infrastructure for next generation care delivery.
- api
- healthcare
- open-source
- soc2
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
Polo travels through your database and creates sample snapshots so you can work with real world data in development.
- database
- snapshot
- productivity
- activerecord
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
- go
- data-integration
- etl
- data-pipeline
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
Amazon Redshift Database Loader implemented in AWS Lambda
SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!
- code-generator
- etl
- dotnet-core
- dotnet
🌱 The ultimate solution for populating your MongoDB database.
Distributed Data Transfer Service for MySQL
- mysql
- data-transfer
- data-migration
- replication
The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
Import, validate, query, and export GTFS Schedule and Realtime data with SQLite, PostgreSQL, and MySQL.
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
The data platform for manufacturing
- oee
- iot
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
- spark
- neo4j-connector
- cypher
- neo4j-driver
The data transformation tool the open-source community deserves.
- data-engineering
- duckdb
- data-lineage
- data-pipeline
Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.
Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰
- csv-import
- data-import
- data-mapping
- data-review