A Python module to bypass Cloudflare's anti-bot page.
- cloudflare
- anti-bot-page
- protected-page
- scrape
Repository Index
47,133 repositories indexed
49–72 of 127 · Most stars
A browser testing and web crawling library for PHP and Symfony
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
- datadome
- perimeterx
- akamai
- antidetect-browser
Scrape Twitter for Tweets
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Get info from any web service or page
- opengraph
- twitter-cards
- embeds
- scraping
Find web directories without bruteforce
PufferPanel is an open source game server management panel, designed for both small networks and personal use
- game-server
- minecraft
- srcds
- panel
Async Python 3.6+ web scraping micro-framework based on asyncio
- asyncio
- aiohttp
- asyncio-spider
- crawler
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
- fredy
- immowelt
- appartment
- appartments
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
Simple web scraping for R
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
A chat bot for Slack (https://slack.com).
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
- meta-search-engine
- rust
- search-engine
- fast
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
Crawly, a high-level web crawling & scraping framework for Elixir.
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
- agpl
- ai-agents
- anti-bot
- crawling
A high performance web crawler / scraper in Elixir.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Java API For Chrome and Firefox