Scrapy, a fast high-level web crawling & scraping framework for Python.
- python
- scraping
- crawling
- framework
Repository Index
71,372 visible repositories in the index
1–24 of 37 · Most stars
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
A next-generation crawling and spidering framework.
- crawler
- web-spider
- gocrawler
- spider-framework
API Support for your favorite torrent trackers
A little like that j-thing, only in Go.
- selector-strings
- jquery
- html-parsing
- goquery
Get the main content of any page as Markdown.
The RSS feed for websites missing it
- php
- rss-bridge
- unlicense
- atom-feed
The Unofficial TikTok API Wrapper In Python
- tiktok-api
- python
- api
- trending
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
Jobs scraper library for LinkedIn, Indeed, Glassdoor, ZipRecruiter, Bayt, Naukri, BDJobs & HelloWork
- indeed
- ziprecruiter
- jobs-scraper
An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.
- ai-agent
- automation
- data-extraction
- event-driven-architecture
Downloads albums from the web in bulk for archive purposes
- ripme
- java
- album
- downloader
job-ops: DevOps principles applied to job hunting. A self-hosted pipeline to track, analyze, and assist your application process
- automation
- self-hosted
- job-search
- jobhunt
freehire — the open-source search engine for job seekers
- ats
- careers
- hiring
- job-aggregator
Use Google News Scraper API to obtain the latest global news for your project, including a wide range of sources, headlines, URLs, and publication dates from the Google News platform.
- google-news-scraper
- google-scraper
- google-scraper-api
- news-scraper
Scrape Facebook public pages without an API key
- scraping
- facebook-scraping
- facebook-scraper
A browser testing and web crawling library for PHP and Symfony
DecryptLogin: APIs for loginning some websites by using requests.
- requests
- login
- python3
- spider
Stock market analyzer and predictor using Elasticsearch, Twitter, News headlines and Python natural language processing and sentiment analysis
- stock-market
- stock-prediction
- python
- tweepy
多播放源自动采集在线影视网站 golang vue
- axios
- elementplus
- film
- gin
HTML parsing and querying with CSS selectors
- hacktoberfest
- rust
🕷️Github China/Global User Ranking, Global Warehouse Star Ranking (Github Action is automatically updated daily).