TranslatedA Python web scraping tutorial series, learning Python web scraping from 0 to 1, including browser packet capture and mobile APP packet capture with tools such as fiddler and mitmproxy, the use of the various modules involved in web scraping, such as requests, beautifulSoup, selenium, appium, scrapy, etc., as well as IP proxies, CAPTCHA recognition, using Mysql and MongoDB databases with Python, multi-threaded and multi-process scraping, reverse engineering and cracking of CSS scraping encryption, JS scraping reverse engineering, distributed scraping, practical scraping project examples, etc.
- python3
- python-script
- python-spider
Translated:orange_book: Chinese Xinhua Dictionary database. Includes xiehouyu (two-part allegorical sayings), idioms, words, and Chinese characters.
- data
- scraper
- chinese-traditional
- python3
TranslatedThe essence edition of getting started with web crawlers in Python
TranslatedPython web crawler tutorial that takes you from zero to one, covering JS reverse engineering, Selenium, Tesseract OCR recognition, the use of MongoDB, and the Scrapy framework
Python爬虫实战 - 模拟登陆各大网站 包含但不限于:滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝,如果喜欢请start ❤️
Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.
- google
- google-trend
- google-trend-api
- google-trend-strategy
A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.
- google-scholar
- google-scholar-scraper
- python
- python-scraper
A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.
- amazon
- amazon-scraper
- api
- python
Learn how to build your own Google Jobs scraper that simultaneously scrapes Google Jobs for multiple search queries and geo-locations with Python and Oxylabs’ Google Jobs Scraper API. https://oxylabs.io/blog/how-to-scrape-google-jobs
- google-scraping
- python
- scraper
- scraper-api
The process of handling CAPTCHA when extracting public data from Amazon with Oxylabs Amazon Scraper API.
- amazon
- amazon-scraper
- amazon-scraper-api
- captcha
Example end to end data engineering project.
Use Web Scraper API to extract data from Google Finance, including stock titles, pricing, and price changes in percentages.
- finance-api
- financial-data-extraction
- google-finance
- google-finance-api
This is a sample Scrapy project for educational purposes
101 real world web scraping exercises in Python 3 for data journalists
python爬虫
- python
- python3
- urllib
- requests
First ever tool to view "Instagram private posts" anonymously
scrapy中文翻译文档
660+ muapi-hosted generative-media models plus community-submitted third-party API tools (SEO, enrichment, social, scraping) — one YAML file per entry, browsable by capability.
Collection of scripts corresponding to LucidProgramming YouTube tutorials
- python-tutorial
- web-scraping
- youtube-tutorial
- lucidprogramming
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
- sraping
- scrapper
- futures
- futures-historical-data
Universal Reddit Scraper - Works on any Subreddit or User