47,133 repositories indexed
1–20 of 20 · Most stars
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Python ProxyPool for web spider
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Turn any website into a structured API. Extract, automate, search and monitor the web.
A next-generation crawling and spidering framework.
Browser Automation, End-to-End Testing, and Stealth in one library. Bypass bot-detection/CAPTCHAs with SeleniumBase CDP Mode.
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
A Chrome DevTools Protocol driver for web automation and scraping.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Dark Web OSINT Tool
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app
Scrapy project for data capture of vgchartz
Search the web and primary sources, read full pages, and check every citation. MCP server for Claude, Cursor, Codex and more. Runs on your machine.
Code to download, process, and analyse the Online Encyclopedia of Integer Sequences
Contest parser for codeforces.com
A PHP project to check if websites are linking to your website
A package to fetching the maker profile - Super Mario Maker