Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Repository Index
47,133 repositories indexed
1–24 of 26 · Most stars
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.
- dataprep
- data-science
- datapreparation
- dataconnector
Apache Spark to Apache Cassandra connector
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
The MongoDB Spark Connector
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
- go
- data-integration
- etl
- data-pipeline
SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!
- code-generator
- etl
- dotnet-core
- dotnet
The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
- spark
- neo4j-connector
- cypher
- neo4j-driver
Apache HBase Connectors
- database
- java
- hbase
AtlasMap project repository
- atlasmap
📜 A CLI toolkit for extracting and working with your digital history
- chronicle
- chronicle-etl
- personal-data
- etl
A plugin to push to Algolia based on graphQl queries
- gatsby
- algolia
- search
- static-site-generator
Scalable Full History API Solution for Antelope (former EOSIO) based blockchains
- antelope
- api
- big-data
- blockchain
Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.
A high performance data transfer tool that can be used to transfer data from Bitcoin Core blockchain files to a SQL Server database
Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.
Track Go benchmark performance over time by storing results in InfluxDB