Repository Index

Open Data Platform for analysts, quants and AI agents.

• python
• machine-learning
• finance
• stocks

Graph-Native Infrastructure for Context and Accountable AI Systems

• ai
• ai-governance
• artificial-intelligence
• context-engineering

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

• dbt-viewpoint
• slack
• pypa
• data-modeling

OpenRefine is a free, open source power tool for working with messy data and improving it

• datacleansing
• data-analysis
• java
• opendata

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

• data-integration
• high-performance
• offline
• real-time

🧙 Build, run, and manage data pipelines for integrating and transforming data.

• machine-learning
• artificial-intelligence
• data
• data-engineering

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

• data-analysis
• data-science
• large-language-models
• llm

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow

Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.

• dataprep
• data-science
• datapreparation
• dataconnector

Data processing & ETL framework for Ruby

• etl
• etl-ruby
• ruby
• data

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

visual data prep powered by python

• data
• data-pipelines
• etl
• structured-data

Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.

• data-science
• data-visualization
• visualization
• data-analysis

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream

Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰

• csv-import
• data-import
• data-mapping
• data-review

A decision-model-based ontology & knowledge-graph builder — builds governed knowledge graphs from your documents, every fact gated and traceable to its evidence.

• graphrag
• information-extraction
• knowledge-graph
• llm

High performance Kafka consumer for InfluxDB. Supports collectd message formats.

• kafka-influxdb
• kafka-consumer
• logstash
• influxdb

Event-driven OneLake pipeline with automated Upsert merge and Python reporting

• activator
• data-engineering
• data-factory
• data-pipeline

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

A high performance data transfer tool that can be used to transfer data from Bitcoin Core blockchain files to a SQL Server database

• bitcoin

This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.

• magento
• import
• products
• categories

Code to download, process, and analyse the Online Encyclopedia of Integer Sequences