An open-source engine that turns continuous content streams into search and monitoring. Durable ingestion, hybrid search, alerts, and plugins for formats, models and business rules.
Repository Index
47,133 repositories indexed
1–16 of 16 · Most stars
Replace 'hub' with 'ingest' in any GitHub URL to get a prompt-friendly extract of a codebase
- ai
- code
- ingestion
- developer-tool
Open Source Metering and Usage Based Billing API ⭐️ Consumption tracking, Subscription management, Pricing iterations, Payment orchestration & Revenue analytics
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
Concurrent and multi-stage data ingestion and data processing with Elixir
- elixir
- genstage
- data-ingestion
- data-processing
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
- data
- ingestion
- replication
- management
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
Fastest end-to-end CSV ingestion for Ruby (with C acceleration). SmarterCSV auto-detects formats, applies smart defaults, and returns Rails-ready hashes for seamless use with ActiveRecord, Sidekiq, parallel jobs, and S3 pipelines — even for messy user-uploaded real-world data.
- csv
- csv-converter
- csv-export
- csv-file
Sentry event forwarding and ingestion service.
- tag-production
Data ingestion for Amazon Elasticsearch Service from S3 and Amazon Kinesis, using AWS Lambda: Sample code
High throughput data ingestion logger to Fluentd, AWS S3 and Treasure Data
- fluentd
- logger
- treasuredata
- aws-s3
Second Brain automation for Obsidian vaults — entity management, ingestion, compression, and sync via Claude Code skills
- ai
- claude-code
- claude-code-plugin
- claude-code-skill
Malware/IOC ingestion and processing engine
Run DuckDB inside Excel VBA: pandas-style analytics via in-process SQL + native C bridge. High-speed Range/Array ingestion, Parquet/CSV/JSON I/O.
Data ingestion engine — Singer taps to Parquet in cloud buckets