Sign in

Repository Index

142,273 visible repositories in the index

Tags: ETL
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 36 · Most stars

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

  • bigquery
  • copy-database
  • data-ingestion
  • data-integration

The Data Transfer Project makes it easy for platforms to build interoperable user data portability features. We are establishing a common framework, including data models and protocols, to enable direct transfer of data both into and out of participating online service providers.

  • data
  • transfer
  • portability
  • data-portability

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。

  • mongo-database
  • mongodb-oplog
  • replication
  • active-active

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

  • analytics
  • bigquery
  • data-modeling
  • data-pipelines

Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.

  • apis
  • data
  • rust
  • sql

Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.

  • integration
  • reactive
  • akka
  • akka-stream

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.