Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–10 of 10 · Most stars
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
Crawl a site to generate knowledge files to create your own custom GPT from a URL
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Self-hosted web and AI crawler analytics for Generative Engine Optimization (GEO). Track visitors, AI search referrals, crawl logs, content coverage, and conversions. Jointly published by 猎看科技 · 北京日报社智源GEO. MIT licensed.
- ai-analytics
- geo
- mit
- postgresql
A Payload CMS 3 plugin for Auth.js/NextAuth 5
- payload-plugin
- payloadcms
- authjs
- next-auth
a dungeon crawler written in TypeScript using React and svg
Practical pinned gist which show your latest medium status 📌
- awesome-pinned-gists
- medium-stats
- github-gists
- github-action
No description provided by the repository host.