Sign in

Repository Index

94,845 visible repositories in the index

Tags: Web scraper
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 42 · Most stars

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

📊 Blazing fast Python framework for web crawling, testing, scraping, and stealth. Supports pytest. Bypasses bot-detection with CDP Mode. Compatible with Playwright.

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

This is a python API which allows you to get the transcript/subtitles for a given YouTube video. It also works for automatically generated subtitles and it does not require an API key nor a headless browser, like other selenium based solutions do!

TranslatedINFO-SPIDER is a crawler toolbox 🧰 that brings many data sources together in one place, designed to help users get their own data back safely and quickly. The tool's code is open source and the process is transparent. Supported data sources include GitHub, QQ Mail, NetEase Mail, Alibaba Mail, Sina Mail, Hotmail, Outlook Mail, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ groups, generating a Moments (WeChat friend circle) photo album, browser browsing history, 12306, cnblogs (博客园), CSDN blog, OSChina blog, Jianshu.

SwiftSoup: Pure Swift HTML Parser, with best of DOM, CSS, and jquery (Supports Linux, iOS, Mac, tvOS, watchOS)

  • swift
  • swiftsoup
  • parse
  • html-document

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Utilities for downloading the EPG (Electronic Program Guide) for thousands of TV channels from hundreds of sources.

AutoCLI is a Blazing fast, memory-safe command-line tool — Fetch information from any website with a single command. Covers Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, and 55+ sites, with support for controlling Electron desktop apps, integrating local CLI tools (gh, docker, kubectl), now powered by AutoCLI.ai .