🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Repository Index
47,133 repositories indexed
1–14 of 14 · Most stars
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
AKShare is an elegant and simple financial data interface library for Python, built for human beings! 开源财经数据接口库
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers catalog.data.gov, open.canada.ca/data, data.humdata.org among many other sites.
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Clean APIs for data cleaning. Python implementation of R package Janitor
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Build, test, deploy, iterate - Dev and prod tool for data science pipelines
:chart_with_upwards_trend: A live cryptocurrency historical trade data blotter. Download live historical trade data from any cryptoexchange, be it for machine learning, backtesting/visualizing trading strategies or for Quantopian/Zipline.
Stock price prediction using a deterministic algorithm inspired by LSTM, focusing on pattern recognition in historical data.
Data ingestion engine — Singer taps to Parquet in cloud buckets