47,133 repositories indexed
1–24 of 26 · Most stars
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Crawl a site to generate knowledge files to create your own custom GPT from a URL
Turn any website into a structured API. Extract, automate, search and monitor the web.
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
The RSS feed for websites missing it
Fast website link checker in Go
Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.
WKZombie is a Swift framework for iOS/OSX to navigate within websites and collect data without the need of User Interface or API, also known as Headless browser. It can be used to run automated tests / snapshots and manipulate websites using Javascript.
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
Crawly, a high-level web crawling & scraping framework for Elixir.
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
A AI Browser run any site JavaScript from Python with the real Firefox engine. Headless SpiderMonkey + Necko for automation and scraping.
Extract structured data from web sites. Web sites scraping.
Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
Scrape any website, article or RSS/Atom Feed with ease!
Mac app for website auditing and crawling
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.
Uses parselets and rwget to generate csv files from websites
Website to Scrapping all writeup from http://ctftime.org/ and you can organize which to read first!
Go package for scraping website recipes
A PHP project to check if websites are linking to your website