Repository Index

Open Data Platform for analysts, quants and AI agents.

• python
• machine-learning
• finance
• stocks

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering

🧙 Build, run, and manage data pipelines for integrating and transforming data.

• machine-learning
• artificial-intelligence
• data
• data-engineering

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

• ai-agents
• data-engineering
• document-ai
• generative-ai

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

• data-analysis
• data-science
• large-language-models
• llm

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake

A system for agentic LLM-powered data processing and ETL

• data
• etl
• llm
• python

Cartography is a Python tool that pulls infrastructure assets and their relationships into a Neo4j graph database.

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow

The open source repository for Electricity Maps data parsers that powers the world's most comprehensive electricity data platform

• sustainability
• data-visualization
• climate-change
• hacktoberfest

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

• dataops
• dataops-platform
• elt
• open-source

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

The data transformation tool the open-source community deserves.

• data-engineering
• duckdb
• data-lineage
• data-pipeline

High performance Kafka consumer for InfluxDB. Supports collectd message formats.

• kafka-influxdb
• kafka-consumer
• logstash
• influxdb

Real-time GTFS database loader and ORM library.

A library and supporting script for pulling data from vSphere and inserting it into InfluxDB