The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
- amazon
- amazon-scraper
- python
- web-scraper
Repository Index
78,850 visible repositories in the index
145–168 of 328 · Most stars
Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Scrapy+Splash for JavaScript integration
- scrapy
- headless-browsers
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
- automation
- content-monitoring
- crawler
- douyin
🤖 Top-rated tools to scrape all major public sections from Facebook, Instagram, and Twitter (X) including posts (likes/comments), photos/videos, contact information, followers, following and much more.
- facebook-page-insights
- instagram-followers-scraper
- facebook-ads-metrics
- facebook-contact-info
A service daemon to run Scrapy spiders
A browser testing and web crawling library for PHP and Symfony
Make your job hunt easy by automating your application process with this Auto Applier
- python
- selenium
- webscraping
- automation
Perplexity Scraper Track brand mentions, analyze rankings, and gain competitor intelligence from Perplexity. Get started in minutes.
- ai-scraper
- llm-api
- llm-scraper
- llm-scraping
A python program that turns an LLM, running on Ollama, into an automated researcher, which will with a single query determine focus areas to investigate, do websearches and scrape content from various relevant websites and do research for you all on its own! And more, not limited to but including saving the findings for you!
历史学习项目：Spark 电影推荐、爬虫、Web 与后台。新版持续开发见 cinematch-recommender。
- spark-mllib
- spark-streaming
- ssm-maven
- scrapy
The Python Code Tutorials
- python
- python3
- scapy
- ethical-hacking
Node.js scraper to get data from Google Play
Tweets metadata scraper & activity analyzer
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
- scraper
- twitter-api
- twitter-bot
媒体服务器演员头像仓库
Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.
- google-trend
- google-trend-api
- google-trend-strategy
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
- search-engine
- python
- scraping
- crawler
python parser for human readable dates
- hacktoberfest
Foundational low latency web data collecting in Rust
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
A curated collection of social media scraping APIs and tools for Instagram, LinkedIn, Twitter/X, TikTok, YouTube, Facebook, and more. Extract posts, profiles, videos, comments, and engagement metrics.
- facebook-api
- facebook-scraping
- linkedin-bot
- linkedin-scraping
admin ui for scrapy/open source scrapinghub
- scrapy
- dashboard
- scrapyd
- scrapy-ui