Apache NiFi
- nifi
- java
- hacktoberfest
- apache
Repository Index
57,138 visible repositories in the index
25–48 of 156 · Most stars
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
- data
- python
- data-engineering
- data-lake
TranslatedDataX integrated with a visual page: select a data source and generate a data synchronization task with one click. Supports data sources such as RDBMS, Hive, HBase, ClickHouse, and MongoDB, and batch creation of RDBMS data synchronization tasks. Integrates an open-source scheduling system, and supports distributed and incremental data synchronization, real-time viewing of running logs, monitoring of executor resources, KILL of running processes, encryption of data source information, and more.
Neo4j graph construction from unstructured data using LLMs
- data-import
- genai
- graph
- graph-rag
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
- data-integration
- clickhouse
- golang
- bigquery
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
- rag
- chonkie
- chunker
- chunking-algorithm
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
Sync data from one Postgres database to another
LFX Community Data Platform (CDP)
- community
- cdp
- open-source
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Database Subsetting and Relational Data Browsing Tool.
Self-hosted platform to unify wearable health data through one AI-ready API.
- wearables
- apple-health
- apple-watch
- fitness
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
sq data wrangler
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- sql
- redpanda
- single-binary
Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.
- dataprep
- data-science
- datapreparation
- dataconnector
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
- data
- ingestion
- replication
- management
Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.
Connect your company's data, let an agent draft an ontology over it, and load it into HelixDB as one graph you can explore and ask questions of. Hosted at foundry.helix-db.com, or run it on your own machine.
- ai-agents
- data-integration
- golang
- graph-database
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
Apache Spark to Apache Cassandra connector