Async Python 3.6+ web scraping micro-framework based on asyncio
- asyncio
- aiohttp
- asyncio-spider
- crawler
142,273 visible repositories in the index
241–264 of 627 · Most stars
Async Python 3.6+ web scraping micro-framework based on asyncio
Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.
In this tutorial, we showcase how to scrape public Google data with Python and Oxylabs API.
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
Free, open-source anti-detect browser launcher for web scraping and multi-accounting. By the ShardX team (formerly ProxyShard). Engine-level fingerprint spoofing in Chromium 158 (WebGL / WebGPU / Client Hints / fonts / TLS), 220+ device profiles bundled, stable QUIC + WebRTC over SOCKS5.
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
A Telegram Mass Surveillance Bot in Python
A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.
Completely free and open-source human-like Instagram bot. Powered by UIAutomator2 and compatible with basically any Android device 5.0+ that can run Instagram - real or emulated.
Random proxy middleware for Scrapy
Twitter scraper and X scraper for Python: scrape tweets, profiles, followers, following and trends without the official API. Twitter scraping library with multi-account pooling, proxy support and async, or a hosted version on Apify.
This code is used to perform web scraping and data extraction from Google Maps. It is particularly designed for obtaining information about businesses, including their name, address, website, phone number, reviews, and more.
LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
Scrapy project to scrape public web directories (educational) [DEPRECATED]
General Assembly's 2015 Data Science course in Washington, DC
Self-hosted web search for AI agents and SEO. Google, Bing, Yandex, Baidu, DuckDuckGo & Ecosia, rendered in a real browser: ranked results and clean page Markdown. API, CLI, MCP 🎉
Undetected Python version of the Playwright testing and automation library.
Information Gathering Instagram.
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
Free HTTP, SOCKS4 and SOCKS5 proxy lists, re-checked every hour. Plain text and JSON, sorted fastest first, with response time, exit IP, ASN and geolocation. No signup and no API key.
The web browser built for scraping
Get clean data from tricky documents, powered by vision-language models ⚡
A highly efficient, fast, powerful and light-weight anime downloader and streamer for your favorite anime.