Repository Index

Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.

• camel
• integration
• java
• microservices

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

• bigquery
• copy-database
• data-ingestion
• data-integration

Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.

• dataprep
• data-science
• datapreparation
• dataconnector

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.

• iot
• iot-gateway
• gateway
• mqtt

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

• analytics
• bigquery
• data-modeling
• data-pipelines

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

• change-data-capture
• data
• data-collection
• data-engineering

Polo travels through your database and creates sample snapshots so you can work with real world data in development.

• database
• snapshot
• productivity
• activerecord

📜 A CLI toolkit for extracting and working with your digital history

• chronicle
• chronicle-etl
• personal-data
• etl

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.

🇪🇺 Warehouse-native Analytics compatible with Google Analytics and Matomo tracking protocols. Ingest into ClickHouse, BigQuery or CSV/Parquet on S3/GCS/local fs while maintaining complete control over your data.

• clickhouse
• ga4
• tracker
• bigquery

Enables search with Algolia in Hugo static sites

Read JMeter results in a csv file and put results in Influxdb database and allows you to create beautiful graphs with Grafana

• jmeter
• influxdb
• grafana
• perfmon

Jmeter JTL parsing with Logstash for Elasticseacrh and Influxdb ...

• jmeter
• logstash
• filebeat
• elasticsearch