Trending repositories

1.cocoindex-io / cocoindex

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python
• +14 more

2.dbt-labs / dbt

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

• dbt-viewpoint
• slack
• pypa
• data-modeling
• business-intelligence
• analytics
• +1 more

3.Unstructured-IO / unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp
• ocr
• information-retrieval
• +14 more

4.dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake
• data-loading
• data-warehouse
• +4 more

5.apache / seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time
• apache
• batch
• +8 more

6.apache / nifi

Apache NiFi

• nifi
• java
• hacktoberfest
• apache

7.bruin-data / bruin

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines
• python
• snowflake
• +5 more

8.apache / camel

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices
• cloud-native
• data-transformation
• +10 more

9.timeplus-io / proton

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda
• single-binary
• stream-processing
• +10 more

10.thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.

• iot
• iot-gateway
• gateway
• mqtt
• opc-ua
• thingsboard
• +11 more

11.Wisser / Jailer

Database Subsetting and Relational Data Browsing Tool.

• database
• subsetting
• frontend
• export
• extract
• sql
• +13 more

12.snowplow / snowplow

The leader in Customer Data Infrastructure

• snowplow
• analytics
• data
• data-pipeline
• data-collection
• product-analytics
• +3 more

13.arkflow-rs / arkflow

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream
• sql
• datafusion
• +14 more

14.united-manufacturing-hub / united-manufacturing-hub

The data platform for manufacturing

• oee
• iot

15.quixio / quix-streams

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering
• data-science
• machine-learning
• +8 more

16.linuxfoundation / crowd.dev

LFX Community Data Platform (CDP)

• community
• cdp
• open-source

17.aws / aws-sdk-pandas

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow
• apache-parquet
• data-engineering
• +14 more

18.ConduitIO / conduit

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

• go
• data-integration
• etl
• data-pipeline
• data-engineering
• data-stream
• +3 more

19.apache / inlong

Apache InLong - a one-stop, full-scenario integration framework for massive data

• inlong
• one-stop-service
• data-streaming
• event-streaming
• framework
• massive-data-integration
• +1 more

20.apache / gobblin

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

• data
• ingestion
• replication
• management
• apache

21.apache / cassandra-spark-connector

Apache Spark to Apache Cassandra connector

• cassandra
• scala
• spark

22.pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)

• cdc
• tidb
• mysql
• kafka
• ticdc
• dm

23.mongodb / mongo-spark

The MongoDB Spark Connector

• spark
• spark-packages
• mongo-spark
• mongodb
• connector

24.Breeze0806 / go-etl

go-etl is a toolset for data extraction, transformation and loading.

• etl
• mysql
• go
• postgresql
• greenplum
• csv
• +5 more

25.embulk / embulk

Embulk: Pluggable Bulk Data Loader.

• embulk
• bulk-loader

26.neo4j / neo4j-spark-connector

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs

• spark
• neo4j-connector
• cypher
• neo4j-driver
• bolt
• hacktoberfest

27.elastic / elasticsearch-hadoop

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

28.pinterest / secorArchived

Secor is a service implementing Kafka log persistence

• kafka

29.PatMartin / Dex

Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.

• data-science
• data-visualization
• visualization
• data-analysis
• data-mining
• java
• +7 more

30.atlasmap / atlasmap

AtlasMap project repository

• atlasmap

31.Stratio / Spark-MongoDBArchived

Spark library for easy MongoDB access