Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
API Support for your favorite torrent trackers
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
Turn any webpage into structured data using LLMs
The Unofficial TikTok API Wrapper In Python
- tiktok-api
- python
- api
- trending
Distributed crawler powered by Headless Chrome
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
SwiftSoup: Pure Swift HTML Parser, with best of DOM, CSS, and jquery (Supports Linux, iOS, Mac, tvOS, watchOS)
- swift
- swiftsoup
- parse
- html-document
Jobs scraper library for LinkedIn, Indeed, ZipRecruiter, Glassdoor, SEEK, Naukri, HelloWork, Bayt & BDJobs
- indeed
- linkedin
- ziprecruiter
- linkedin-scraper
Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot
- python
- bot
- client
- python3
An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.
TranslatedIPProxyPool proxy pool project, providing proxy IPs
job-ops: DevOps principles applied to job hunting. A self-hosted pipeline to track, analyze, and assist your application process
Downloads albums from the web in bulk for archive purposes
- ripme
- java
- album
- downloader
Shadow any website for offline viewing, with the JavaScript stripped out
AutoCLI is a Blazing fast, memory-safe command-line tool — Fetch information from any website with a single command. Covers Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, and 55+ sites, with support for controlling Electron desktop apps, integrating local CLI tools (gh, docker, kubectl), now powered by AutoCLI.ai .
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
虎牙,斗鱼,抖音,BiliBili,TikTok,Twitch🔥热门🔥智能直播视频剪辑发布AI机器人,自动化🤖,全智能化⚙(智能生成切片,标题,封面,简介),可视化👓,平台热门监控🌡,丰富插件随意扩展🕹,快速部署⚡,视频账号打造自动发布🌟,支持DIY🎮
Export tweets, bookmarks, lists and much more from Twitter(X) web app. (推文/书签/收藏/列表导出工具)
- twitter
- archiver
- exporter
- twitter-api
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Generate and download e-books from online sources.