🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python scraper based on AI
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
An open source library for deep learning end-to-end dialog systems and chatbots.
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
新闻网页正文通用抽取器 Beta 版.
- python3
- webcrawler
- webspider
Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
JSFinder is a tool for quickly extracting URLs and subdomains from JS files on a website.
Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python package for scraping recipes data
- data-extraction
- food
- html-parsing
- json-ld
A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.
- google-scholar
- google-scholar-scraper
- python
- python-scraper
A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.
- amazon
- amazon-scraper
- api
- python
Financial Data Extraction from Investing.com with Python
- investing-api
- historical-data
- investing
- investing-python
A robust, all-in-one GPT interface for Discord. ChatGPT-style conversations, image generation, AI-moderation, custom indexes/knowledgebase, youtube summarizer, and more!
This code is used to perform web scraping and data extraction from Google Maps. It is particularly designed for obtaining information about businesses, including their name, address, website, phone number, reviews, and more.
- chromium
- data
- data-scraping
- google
LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping
- webscraping
- python3
- osint
- enumeration
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.
🤖 Scrape data from HTML websites automatically by just providing examples
Lightweight library for scraping web-sites with LLMs