An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
- etl-pipeline
- etl-framework
- spark
- apache-spark
95,842 visible repositories in the index
49–72 of 86 · Most stars
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Apache InLong - a one-stop, full-scenario integration framework for massive data
Elasticsearch File System Crawler (FS Crawler)
Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.
Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync
visual data prep powered by python
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
Distributed Data Transfer Service for MySQL
Import, validate, query, and export GTFS Schedule and Realtime data with SQLite, PostgreSQL, and MySQL.
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
📜 A CLI toolkit for extracting and working with your digital history
go-etl is a toolset for data extraction, transformation and loading.
A plugin to push to Algolia based on graphQl queries
Event-driven OneLake pipeline with automated Upsert merge and Python reporting
RDF-Centric Map/Reduce Framework and Freebase data conversion tool
Analyse GTFS Schedule data by importing it into a PostgreSQL database.
Scalable Full History API Solution for Antelope (former EOSIO) based blockchains
Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.
Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.
This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.