Repository Index

Python scraper based on AI

• scraping
• scraping-python
• llm
• web-crawler

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

• web-scraping
• web-crawling
• npm
• headless-chrome

Weighs the soul of incoming HTTP requests to stop AI crawlers

• defense
• security

TranslatedSome interesting examples of python crawlers that are friendly to beginners. Mainly crawls websites such as 淘宝 (Taobao, "searching for treasure"), 天猫 (Tmall, "sky cat"), 微信 (WeChat, "micro message"), 微信读书 (WeChat Reading), 豆瓣 (Douban, "bean petal"), and QQ.

• crawler
• spider
• taobao
• tmall

UAParser.js: The Essential Web Development Tool for User-Agent Detection. Detect Browsers, OS, Devices, Bots, Apps, AI Crawlers, and more. Run in Browser or Node

• user-agent-parser
• browser-detection
• user-agent
• client-hints

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

• php
• user-agent
• crawler
• spider

teler-waf is a Go HTTP middleware that protects local web services from OWASP Top 10 threats, known vulnerabilities, malicious actors, botnets, unwanted crawlers, and brute force attacks.

• ids
• teler
• waf
• teler-ids

:vertical_traffic_light: Block malicious crawlers with Cloudflare Firewall Rules

• cloudflare
• firewall
• user-agent
• firewall-rules

Voight-Kampff is a Ruby gem that detects bots, spiders, crawlers and replicants

Astray is a lua based maze, room and dungeon generation library for dungeon crawlers and rougelike video games

• lua
• maze
• room
• rougelike