Repository Index

A next-generation crawling and spidering framework.

• crawler
• web-spider
• gocrawler
• spider-framework

Dark Web OSINT Tool

• tor
• tor-network
• deepweb
• dark-web

Use Google News Scraper API to obtain the latest global news for your project, including a wide range of sources, headlines, URLs, and publication dates from the Google News platform.

• google-news-scraper
• google-scraper
• google-scraper-api
• news-scraper

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

• java
• webcrawling
• warc
• heritrix

Gospider - Fast web spider written in Go

• spider
• crawler
• go
• gospider

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

• c-sharp
• abot
• abot-nuget
• crawler

python爬虫项目合集，从基础到js逆向，包含基础篇、自动化篇、进阶篇以及验证码篇。案例涵盖各大网站(xhs douyin weibo ins boss job，jd...)，你将会学到有关爬虫以及反爬虫、自动化和验证码的各方面知识

• playwright
• python
• python-crawler
• reverse-engineering

Scrapy project to scrape public web directories (educational) [DEPRECATED]

Scrapy project for data capture of vgchartz

A PHP project to check if websites are linking to your website

• php
• competitors
• seo
• seo-optimization