Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.
Repository Index
99,825 visible repositories in the index
1–24 of 145 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Create agents that monitor and act on your behalf. Your agents are standing by!
- automation
- notifications
- scraper
- webscraping
🕵️♂️ Collect a dossier on a person by username from 6K websites
- osint
- social-network
- identification
- socmint
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
- website-monitor
- website-monitoring
- change-detection
- monitoring
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
- anti-detect
- bot-detection
- browser-automation
- chromium
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
Make Any Website into CLI & Use your logged-in browser by AI agent.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Deploy headless browsers in Docker. Run on our cloud or bring your own. Free for non-commercial uses.
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
Capture website screenshots
The RSS feed for websites missing it
- php
- rss-bridge
- unlicense
- atom-feed
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
- golang
- google-maps-scraping
- web-scraper
- web-scraping
✨ The all-contributors bot website and documentation. Recognize all contributors, not just the ones who push code ✨
- contributors
- all-contributors
- opensource
- opensource-management
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
An Efficient ProxyPool with Getter, Tester and Server
💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js
- offline-web-pages
- downloader
- scraper
- assets
Download an entire website from the Wayback Machine.
EyeWitness is designed to take screenshots of websites, provide some server header info, and identify default credentials if possible.
Undetected version of the Playwright testing and automation library.
- automation
- bot
- bots
- botting