🕵️♂️ Collect a dossier on a person by username from 6K websites
- osint
- social-network
- identification
- socmint
Repository Index
64,265 visible repositories in the index
1–24 of 24 · Most stars
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- wget
- browser-bookmarks
- pinboard
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
Python ProxyPool for web spider
API Support for your favorite torrent trackers
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
- apify
- automation
- beautifulsoup
- crawler
One-stop Proxies Crawling and Aggregation Platform
- proxypool
An Efficient ProxyPool with Getter, Tester and Server
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
- high-availability
- scrapy
- ipproxy
- distributed
A library that scrapes Linkedin for user data
- linkedin-scraper
- scraper
- profile
TranslatedIPProxyPool proxy pool project, providing proxy IPs
Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes
- crawler
- proxypool
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
- crawler
- python3
- scylla
- proxy-pool
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。
- emby
- jav-scraper
- jav
- plugin
Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
- amazon-scraper
- amazon-scraping
- e-commerce-api
- web-scraping
v2ray、clash免费节点爬虫，自动更新
Use Google News Scraper API to obtain the latest global news for your project, including a wide range of sources, headlines, URLs, and publication dates from the Google News platform.
- google-news-scraper
- google-scraper
- google-scraper-api
- news-scraper
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
- amazon
- amazon-scraper
- python
- web-scraper
Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
Practical pinned gist which show your latest medium status 📌
- awesome-pinned-gists
- medium-stats
- github-gists
- github-action
Browser extension to simulate browsing behaviour in search engines.
- automation
- audit
- browser-extension
- firefox
The repository host gives no description.