LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
- ai-agents
- data-engineering
- document-ai
- generative-ai
110,088 visible repositories in the index
1–22 of 22 · Most stars
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
Database Subsetting and Relational Data Browsing Tool.
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Extract Transform Load for Python 3.5+
cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
JSON output plugin for changeset extraction
Python Extract Transform and Load Tables of Data
Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.
A decision-model-based ontology & knowledge-graph builder — builds governed knowledge graphs from your documents, every fact gated and traceable to its evidence.
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
📜 A CLI toolkit for extracting and working with your digital history
go-etl is a toolset for data extraction, transformation and loading.
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.