:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Repository Index
85,081 visible repositories in the index
49–72 of 86 · Most stars
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
The simple, stupid batch framework for Java
- java
- batch-processing
- batch
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
🌱 The ultimate solution for populating your MongoDB database.
The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
Import, validate, query, and export GTFS Schedule and Realtime data with SQLite, PostgreSQL, and MySQL.
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
Graph-based security analysis for everyone
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
- spark
- neo4j-connector
- cypher
- neo4j-driver
A decision-model-based ontology & knowledge-graph builder — builds governed knowledge graphs from your documents, every fact gated and traceable to its evidence.
- graphrag
- information-extraction
- knowledge-graph
- llm
AtlasMap project repository
- atlasmap
A plugin to push to Algolia based on graphQl queries
- gatsby
- algolia
- search
- static-site-generator
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
RDF-Centric Map/Reduce Framework and Freebase data conversion tool
Analyse GTFS Schedule data by importing it into a PostgreSQL database.
Scalable Full History API Solution for Antelope (former EOSIO) based blockchains
- antelope
- api
- big-data
- blockchain
This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.
- magento
- import
- products
- categories
Database synchronizer for Magento 2 (and Wordpress), based on Magerun2. Keeping development, staging and production in sync easily.
Flume NG MongoDB source.
- flume
- sink
- mongodb
Code to download, process, and analyse the Online Encyclopedia of Integer Sequences