Turn any webpage into structured data using LLMs
Repository Index
107,098 visible repositories in the index
1–24 of 27 · Most stars
💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js
- offline-web-pages
- downloader
- scraper
- assets
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
TikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.
Google Maps Scraper & Lead Generation Tool. Extract 50+ data points including business emails, phone numbers, and social profiles. Includes enrichment features, API access, and a generous free plan with 200 searches/month.
- google-maps-scraping
- google-maps
- google-maps-api
- google-maps-review
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
- amazon
- amazon-scraper
- python
- web-scraper
Automatically crawl arXiv papers daily and summarize them using AI. Illustrating them using GitHub Pages.
DecryptLogin: APIs for loginning some websites by using requests.
- requests
- login
- python3
- spider
Scrape metadata from any URL using Open Graph, JSON-LD, HTML meta tags, and smart fallbacks.
- metadata
- article
- cheerio
- extract
A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript
- playwright
- playwright-python
- scraping
- screenshot-utility
Stock market analyzer and predictor using Elasticsearch, Twitter, News headlines and Python natural language processing and sentiment analysis
- stock-market
- stock-prediction
- python
- tweepy
A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.
- google-scholar
- google-scholar-scraper
- python
- python-scraper
A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.
- amazon
- amazon-scraper
- api
- python
ZonyLrcToolsX 是一个能够方便地下载歌词的小软件。
- lyrics-fetcher
- taglib
- mp3
- dotnet-core
Pull current and historical baseball statistics using Python (Statcast, Baseball Reference, FanGraphs)
- baseball
- sabermetrics
- python
- data
This code is used to perform web scraping and data extraction from Google Maps. It is particularly designed for obtaining information about businesses, including their name, address, website, phone number, reviews, and more.
- chromium
- data
- data-scraping
👻 Experimental library for scraping websites using OpenAI's GPT API.
- gpt
- webscraping
- openai-api
Get Instagram posts/profile/hashtag data without using Instagram API
- instagram-crawler
- python
- webdriver
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
- python
- lxml
- xpath
- xml
Parse web pages using CSS query selectors
Simple script for downloading Youtube comments without using the Youtube API
- youtube-comments
- youtube
- python
- data-science
DOM Traversing and Scraping using GraphQL
WKZombie is a Swift framework for iOS/OSX to navigate within websites and collect data without the need of User Interface or API, also known as Headless browser. It can be used to run automated tests / snapshots and manipulate websites using Javascript.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json