Repository Index

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python

SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.

TranslatedDataX is the open-source version of Alibaba Cloud DataWorks Data Integration.

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp

Logstash - transport and process your logs, events, or other data

• etl-framework
• streaming
• logging
• java

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

• aws
• gcp
• azure
• sql

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

• data
• ingestion
• replication
• management

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

• etl
• data-integration
• python
• postgresql

Data processing & ETL framework for Ruby

• etl
• etl-ruby
• ruby
• data

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

Apache InLong - a one-stop, full-scenario integration framework for massive data

• inlong
• one-stop-service
• data-streaming
• event-streaming

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

• csv
• parser
• writer
• reader

Polo travels through your database and creates sample snapshots so you can work with real world data in development.

• database
• snapshot
• productivity
• activerecord

The simple, stupid batch framework for Java

• java
• batch-processing
• batch

:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.

• porter
• data-import
• framework
• data-transformation

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

• java
• smooks
• enterprise-integration
• etl

📜 A CLI toolkit for extracting and working with your digital history

• chronicle
• chronicle-etl
• personal-data
• etl

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

RDF-Centric Map/Reduce Framework and Freebase data conversion tool

A Substrate indexing framework

• substrate

Algorand's data pipeline framework.