Sign in

Repository Index

107,098 visible repositories in the index

Tags: Web scraper
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 60 · Most stars

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

  • pocket
  • wget
  • browser-bookmarks
  • pinboard

⬛️ GUI/CLI tool and library for saving complete web pages as a single HTML file

  • save-the-internet
  • procrastination
  • its-mine
  • no-more-404

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

  • python
  • bot
  • client
  • python3

Jobs scraper library for LinkedIn, Indeed, Glassdoor, ZipRecruiter, Bayt, Naukri, BDJobs, HelloWork & SEEK

  • indeed
  • linkedin
  • ziprecruiter
  • linkedin-scraper

Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes

Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era

  • crawler
  • python3
  • scylla
  • proxy-pool

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

  • aitools
  • crawl
  • scrape
  • serp

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

  • java
  • webcrawling
  • warc
  • heritrix

AutoCLI is a Blazing fast, memory-safe command-line tool — Fetch information from any website with a single command. Covers Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, and 55+ sites, with support for controlling Electron desktop apps, integrating local CLI tools (gh, docker, kubectl), now powered by AutoCLI.ai .

Point it at any web page and it finds the video, transcodes it, and streams it to your TV in real time. It even burns in subtitles.